DRL-Based Joint Resource Allocation and Device Orchestration for Hierarchical Federated Learning in NOMA-Enabled Industrial IoT

Federated learning (FL) provides a new paradigm for protecting data privacy in Industrial Internet of Things (IIoT). To reduce network burden and latency brought by FL with a parameter server at the cloud, hierarchical federated learning (HFL) with mobile edge computing (MEC) servers is proposed. However, HFL suffers from a bottleneck of communication and energy overhead before reaching satisfying model accuracy as IIoT devices dramatically increase. In this article, a deep reinforcement learning (DRL)-based joint resource allocation and IIoT device orchestration policy using nonorthogonal multiple access is proposed to achieve a more accurate model and reduce overhead for MEC-assisted HFL in IIoT. We formulate a multiobjective optimization problem to simultaneously minimize latency, energy consumption, and model accuracy under the constraints of computing capacity and transmission power of IIoT devices. To solve it, we propose a DRL algorithm based on deep deterministic policy gradient. Simulation results show proposed algorithm outperforms others.

[1]  Kwang-Cheng Chen,et al.  User Access Control in Open Radio Access Networks: A Federated Deep Reinforcement Learning Approach , 2022, IEEE Transactions on Wireless Communications.

[2]  Jer Shyuan Ng,et al.  Decentralized Edge Intelligence: A Dynamic Resource Allocation Framework for Hierarchical Federated Learning , 2022, IEEE Transactions on Parallel and Distributed Systems.

[3]  Chunyan Miao,et al.  Dynamic Edge Association and Resource Allocation in Self-Organizing Hierarchical Federated Learning Networks , 2021, IEEE Journal on Selected Areas in Communications.

[4]  Zhu Han,et al.  Deep Reinforcement Learning Assisted Federated Learning Algorithm for Data Management of IIoT , 2021, IEEE Transactions on Industrial Informatics.

[5]  Tom H. Luan,et al.  Secure and Efficient Federated Learning for Smart Grid With Edge-Cloud Collaboration , 2021, IEEE Transactions on Industrial Informatics.

[6]  Mehdi Bennis,et al.  Joint Client Scheduling and Resource Allocation Under Channel Uncertainty in Federated Learning , 2021, IEEE Transactions on Communications.

[7]  Shiva Raj Pokhrel,et al.  Multipath Communication With Deep Q-Network for Industry 4.0 Automation and Orchestration , 2021, IEEE Transactions on Industrial Informatics.

[8]  Haomiao Yang,et al.  Efficient and Privacy-Enhanced Federated Learning for Industrial Artificial Intelligence , 2020, IEEE Transactions on Industrial Informatics.

[9]  Shibo He,et al.  Privacy-Preserving Database Assisted Spectrum Access for Industrial Internet of Things: A Distributed Learning Approach , 2020, IEEE Transactions on Industrial Electronics.

[10]  Tony Q. S. Quek,et al.  Multi-Armed Bandit-Based Client Scheduling for Federated Learning , 2020, IEEE Transactions on Wireless Communications.

[11]  Hao Wang,et al.  Optimizing Federated Learning on Non-IID Data with Reinforcement Learning , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[12]  Haijian Sun,et al.  Scheduling Policy and Power Allocation for Federated Learning in NOMA Based MEC , 2020, GLOBECOM 2020 - 2020 IEEE Global Communications Conference.

[13]  Yan Zhang,et al.  Differentially Private Asynchronous Federated Learning for Mobile Edge Computing in Urban Informatics , 2020, IEEE Transactions on Industrial Informatics.

[14]  Xu Chen,et al.  HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learning , 2020, IEEE Transactions on Wireless Communications.

[15]  Dong In Kim,et al.  Toward an Automated Auction Framework for Wireless Federated Learning Services Market , 2019, IEEE Transactions on Mobile Computing.

[16]  H. Vincent Poor,et al.  Performance Optimization of Federated Learning over Wireless Networks , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[17]  Choong Seon Hong,et al.  Energy Efficient Federated Learning Over Wireless Communication Networks , 2019, IEEE Transactions on Wireless Communications.

[18]  Zhisheng Niu,et al.  Device Scheduling with Fast Convergence for Wireless Federated Learning , 2019, ICC 2020 - 2020 IEEE International Conference on Communications (ICC).

[19]  Canh Dinh,et al.  Federated Learning Over Wireless Networks: Convergence Analysis and Resource Allocation , 2019, IEEE/ACM Transactions on Networking.

[20]  Ning Zhang,et al.  A Cloud-Guided Feature Extraction Approach for Image Retrieval in Mobile Edge Computing , 2019, IEEE Transactions on Mobile Computing.

[21]  Deniz Gündüz,et al.  Hierarchical Federated Learning ACROSS Heterogeneous Cellular Networks , 2019, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  H. Vincent Poor,et al.  Scheduling Policies for Federated Learning in Wireless Networks , 2019, IEEE Transactions on Communications.

[23]  Deniz Gündüz,et al.  Federated Learning Over Wireless Fading Channels , 2019, IEEE Transactions on Wireless Communications.

[24]  Deniz Gündüz,et al.  Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[25]  S. H. Song,et al.  Client-Edge-Cloud Hierarchical Federated Learning , 2019, ICC 2020 - 2020 IEEE International Conference on Communications (ICC).

[26]  Albert Y. Zomaya,et al.  Federated Learning over Wireless Networks: Optimization Model Design and Analysis , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[27]  Lucas Santos Dalenogare,et al.  Industry 4.0 technologies: Implementation patterns in manufacturing companies , 2019, International Journal of Production Economics.

[28]  Ke Zhang,et al.  Deep Learning Empowered Task Offloading for Mobile Edge Computing in Urban Informatics , 2019, IEEE Internet of Things Journal.

[29]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[30]  Zhi Ding,et al.  Federated Learning via Over-the-Air Computation , 2018, IEEE Transactions on Wireless Communications.

[31]  Chun-Xia Zhang,et al.  Security-Enhanced User Pairing for MISO-NOMA Downlink Transmission , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[32]  Takayuki Nishio,et al.  Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge , 2018, ICC 2019 - 2019 IEEE International Conference on Communications (ICC).

[33]  Ursula Challita,et al.  Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial , 2017, IEEE Communications Surveys & Tutorials.

[34]  Yi Wu,et al.  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments , 2017, NIPS.

[35]  Shahid Mumtaz,et al.  Massive Internet of Things for Industrial Applications: Addressing Wireless IIoT Connectivity Challenges and Ecosystem Fragmentation , 2017, IEEE Industrial Electronics Magazine.

[36]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[37]  Frank P. Ferrie,et al.  Pareto models for discriminative multiclass linear dimensionality reduction , 2015, Pattern Recognit..

[38]  Peter Richtárik,et al.  Semi-stochastic coordinate descent , 2014, Optim. Methods Softw..

[39]  Shai Shalev-Shwartz,et al.  Stochastic dual coordinate ascent methods for regularized loss , 2012, J. Mach. Learn. Res..

[40]  Thomas D. Burd,et al.  Processor design for portable systems , 1996, J. VLSI Signal Process..

[41]  Tony Q. S. Quek,et al.  Joint Scheduling and Resource Allocation for Hierarchical Federated Edge Learning , 2022, IEEE Transactions on Wireless Communications.

[42]  Yan Zhang,et al.  Communication-Efficient Federated Learning for Digital Twin Edge Networks in Industrial IoT , 2022 .