Multiple Parallel Federated Learning via Over-the-Air Computation

This paper investigates multiple parallel federated learning in cellular networks, where a base station schedules several FL tasks in parallel and each task has a group of devices involved. To reduce the communication overhead, over-the-air computation is introduced by utilizing the superposition property of multiple access channels (MAC) to accomplish the aggregation step. Since all devices use the same radio resource to transfer their local updates to the BS, in order to separate the received signals of different tasks, we use the zero-forcing receiver combiner to mitigate the mutual interference across different groups. Besides, we analyze the impact of receiver combiner and device selection on the convergence of our multiple parallel FL framework. Also, we formulate an optimization problem that jointly considers receiver combiner vector design and device selection for improving FL performance. We address the problem by decoupling it into two sub-problems and solve them alternatively, adopting successive convex approximation (SCA) to derive the receiver combiner vector, and then solve the device scheduling problem with a greedy algorithm. Simulation results demonstrate that the proposed framework can effectively solve the straggler issue in FL and achieve a near-optimal performance on all tasks.

[1]  Yuanming Shi,et al.  Interference Management for Over-the-Air Federated Learning in Multi-Cell Wireless Networks , 2022, IEEE Journal on Selected Areas in Communications.

[2]  K. B. Letaief,et al.  Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications , 2021, IEEE Journal on Selected Areas in Communications.

[3]  Meixia Tao,et al.  Gradient Statistics Aware Power Control for Over-the-Air Federated Learning , 2020, IEEE Transactions on Wireless Communications.

[4]  Vincent K. N. Lau,et al.  Analog Gradient Aggregation for Federated Learning Over Wireless Networks: Customized Design and Convergence Analysis , 2021, IEEE Internet of Things Journal.

[5]  Deniz Gündüz,et al.  One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning , 2020, GLOBECOM 2020 - 2020 IEEE Global Communications Conference.

[6]  Xiaojun Yuan,et al.  Reconfigurable Intelligent Surface Enabled Federated Learning: A Unified Communication-Learning Design Approach , 2020, IEEE Transactions on Wireless Communications.

[7]  Shuguang Cui,et al.  Optimized Power Control for Over-the-Air Federated Edge Learning , 2020, ICC 2021 - IEEE International Conference on Communications.

[8]  Zhisheng Niu,et al.  Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learning , 2020, IEEE Transactions on Wireless Communications.

[9]  Yonina C. Eldar,et al.  UVeQFed: Universal Vector Quantization for Federated Learning , 2020, IEEE Transactions on Signal Processing.

[10]  Jun Zhang,et al.  Communication-Efficient Edge AI: Algorithms and Systems , 2020, IEEE Communications Surveys & Tutorials.

[11]  H. Vincent Poor,et al.  Update Aware Device Scheduling for Federated Learning at the Wireless Edge , 2020, 2020 IEEE International Symposium on Information Theory (ISIT).

[12]  Deniz Gündüz,et al.  One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis , 2020, IEEE Transactions on Wireless Communications.

[13]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[14]  H. Poor,et al.  A Joint Learning and Communications Framework for Federated Learning Over Wireless Networks , 2019, IEEE Transactions on Wireless Communications.

[15]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[16]  Kobi Cohen,et al.  On Analog Gradient Descent Learning Over Multiple Access Fading Channels , 2019, IEEE Transactions on Signal Processing.

[17]  H. Vincent Poor,et al.  Scheduling Policies for Federated Learning in Wireless Networks , 2019, IEEE Transactions on Communications.

[18]  Deniz Gündüz,et al.  Federated Learning Over Wireless Fading Channels , 2019, IEEE Transactions on Wireless Communications.

[19]  Deniz Gündüz,et al.  Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[20]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[21]  Zhi Ding,et al.  Federated Learning via Over-the-Air Computation , 2018, IEEE Transactions on Wireless Communications.

[22]  Kaibin Huang,et al.  Broadband Analog Aggregation for Low-Latency Federated Edge Learning , 2018, IEEE Transactions on Wireless Communications.

[23]  Dan Alistarh,et al.  The Convergence of Sparsified Gradient Methods , 2018, NeurIPS.

[24]  Li Chen,et al.  A Uniform-Forcing Transceiver Design for Over-the-Air Function Computation , 2018, IEEE Wireless Communications Letters.

[25]  Kin K. Leung,et al.  Adaptive Federated Learning in Resource Constrained Edge Computing Systems , 2018, IEEE Journal on Selected Areas in Communications.

[26]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[27]  Alexandros G. Dimakis,et al.  Gradient Coding: Avoiding Stragglers in Distributed Learning , 2017, ICML.

[28]  Kenneth Heafield,et al.  Sparse Communication for Distributed Gradient Descent , 2017, EMNLP.

[29]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[30]  Peter Richtárik,et al.  Federated Optimization: Distributed Machine Learning for On-Device Intelligence , 2016, ArXiv.

[31]  Dan Alistarh,et al.  QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks , 2016, 1610.02132.

[32]  Tao Zhang,et al.  Fog and IoT: An Overview of Research Opportunities , 2016, IEEE Internet of Things Journal.

[33]  Bang Chul Jung,et al.  Opportunistic Function Computation for Wireless Sensor Networks , 2016, IEEE Transactions on Wireless Communications.

[34]  Samy Bengio,et al.  Revisiting Distributed Synchronous SGD , 2016, ArXiv.

[35]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[36]  Slawomir Stanczak,et al.  Harnessing Interference for Analog Function Computation in Wireless Sensor Networks , 2013, IEEE Transactions on Signal Processing.

[37]  Francisco Facchinei,et al.  Decomposition by Partial Linearization: Parallel Optimization of Multi-Agent Systems , 2013, IEEE Transactions on Signal Processing.

[38]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[39]  Slawomir Stanczak,et al.  Robust Analog Function Computation via Wireless Multiple-Access Channels , 2012, IEEE Transactions on Communications.

[40]  Michael Beigl,et al.  Calculation of functions on the RF-channel for IoT , 2012, 2012 3rd IEEE International Conference on the Internet of Things.

[41]  Mark W. Schmidt,et al.  Hybrid Deterministic-Stochastic Methods for Data Fitting , 2011, SIAM J. Sci. Comput..

[42]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[43]  Michael Gastpar,et al.  Computation Over Multiple-Access Channels , 2007, IEEE Transactions on Information Theory.

[44]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[45]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.