Hierarchical Federated Learning for Smart City AIoT Systems

By Soham NandiReviewed by Susha Cheriyedath, M.Sc.Feb 29 2024

In an article published in the journal Future Internet, the researchers introduced multi-level split federated learning (SFL) as a solution for collaborative training of deep learning models in artificial intelligence of things (AIoT) systems in smart cities.

Addressing challenges like data privacy and communication latency, the proposed framework combined split learning (SL) and federated learning (FL). The architecture incorporated message queuing telemetry transport (MQTT) for geographically clustering Internet of things (IoT) devices, utilizing edge and fog computing layers for initial model parameter aggregation.

Background

In response to the increasing pace of urbanization, the concept of smart cities has emerged, leveraging AI and IoT to enhance urban infrastructure and services. Integrating AI and IoT gives rise to AIoT systems, which are capable of analyzing vast amounts of historical and real-time data generated by IoT devices like sensors. This data-driven approach enables more accurate predictions and optimization in various domains, including smart grids, transportation, and healthcare.

As smart cities evolve, the significance of data generated within them transforms into a valuable digital asset. However, the challenge lies in utilizing this data for collaborative learning without compromising data privacy. Traditional machine learning models face limitations in large-scale AIoT systems, especially concerning data privacy, system and statistical heterogeneity, and the scalability of models.

Distributed collaboration machine learning (DCML) approaches such as FL and SL have been proposed to address data privacy challenges by allowing multiple IoT devices (clients) to collaboratively train models without sharing raw local data. FL aggregates model parameters from local models on each client through a central server, while SL splits the deep learning model between the IoT device and the cloud, addressing limited computing resources. However, FL faces challenges in large-scale systems with resource-constrained IoT devices, and SL's performance decreases with an increasing number of clients.

To overcome these limitations, SFL combines FL and SL advantages for parallel data and model training in AIoT systems. Despite being a promising paradigm, SFL efficiency is hindered in large systems due to factors like the number of clients, varying transmission distances, and instability of single cloud center server nodes.

This paper introduced a novel solution – the multi-level SFL architecture – to address the aforementioned gaps in collaborative learning for large-scale AIoT systems. The architecture employed a hierarchical approach with initial model parameter aggregation at edge and fog layers, reducing communication delays and improving scalability. By leveraging the MQTT protocol, geographically clustered IoT devices collaborated, and the inclusion of SL balanced system heterogeneity. 

Proposed Framework

The proposed framework introduced a multi-level SFL in the context of large-scale AIoT systems. This approach combined cloud- and edge-based FL with server-based SL. The architecture involved multiple levels, from a cloud server at the first level to end devices at the Nth level, including edge servers and fog nodes. IoT devices were grouped based on geographic distribution, and each group was assigned a master server. In the Nth level, heterogeneous IoT devices generated data locally using their own data trainers.

Communication utilized the MQTT protocol, facilitating efficient exchange between client nodes and server nodes. The framework reduced communication overhead by using MQTT and enabled collaboration among geographically distributed IoT devices. The SL side involved forward and backward propagations on IoT devices and master servers.

The multi-level FL workflow included the cloud server initiating tasks, edge servers evaluating IoT device resources, and the cloud server randomly initializing global model parameters. Each IoT device updated its local model, and the hierarchy of edge servers aggregated model parameters using the FedAvg algorithm. The framework's iterative process continued until convergence. This multi-level SFL aimed to enhance training efficiency, reduce communication costs, and accommodate a dynamic number of IoT devices in large-scale AIoT systems.

Experimentation

The authors proposed a multi-level SFL framework for simulating real-world smart city AIoT systems in Docker. They evaluated the framework's performance using different datasets (Fashion MNIST, HAM10000) and machine learning models (LeNet, ResNet18). The feasibility was validated by comparing traditional SFL, multi-level FL, and centralized learning in various dataset scenarios.

The experiment was conducted in Docker 24.0.7 and API 1.43, with isolated containers simulating end devices in a smart city AIoT system. Nodes were connected through Docker's bridge network based on their geographical regions. The MQTT protocol and secure socket layer (SSL) sockets ensured secure communication and data transmission.

Two datasets, Fashion MNIST, and HAM10000, were employed to assess the framework's robustness, especially in handling unbalanced datasets. The performance was evaluated based on model accuracy and training time. Multi-level SFL was compared with traditional FL, demonstrating competitive performance and faster convergence.

The impact of varying numbers of clients on model accuracy and training time was explored. Results showed that the convergence speed slightly decreased as the number of clients increased, but multi-level SFL outperformed centralized learning. The researchers also analyzed the effect of different levels in the SFL architecture, indicating that multi-level SFL was superior, especially in non-independent and non-identically distributed scenarios.

Finally, the time cost of multi-level SFL was compared with multi-level FL for both small and large machine learning tasks (LeNet and ResNet18). Multi-level SFL exhibited lower time overhead in large machine-learning tasks, making it more suitable for resource-limited AIoT clients.

Conclusion

In conclusion, the authors introduced a groundbreaking multi-level SFL framework for large-scale AIoT systems, addressing connectivity challenges and enhancing processing speed. The integration of SFL balanced system heterogeneity, ensuring scalability and resilience against single points of failure. Experimental results using MQTT and Docker validated the framework's practical feasibility, demonstrating improved accuracy under large-scale clients. Despite advantages, further exploration is needed for optimizing transmission overhead and balancing communication in varying dataset sizes. Future work will delve into these aspects for comprehensive refinement.