Federated Learning: A signal processing perspective

The dramatic success of deep learning is largely due to the availability of data. Data samples are often acquired on edge devices, such as smart phones, vehicles and sensors, and in some cases cannot be shared due to privacy considerations. Federated learning is an emerging machine learning paradigm for training models across multiple edge devices holding local datasets, without explicitly exchanging the data. Learning in a federated manner differs from conventional centralized machine learning, and poses several core unique challenges and requirements, which are closely related to classical problems studied in the areas of signal processing and communications. Consequently, dedicated schemes derived from these areas are expected to play an important role in the success of federated learning and the transition of deep learning from the domain of centralized servers to mobile edge devices. In this article, we provide a unified systematic framework for federated learning in a manner that encapsulates and highlights the main challenges that are natural to treat using signal processing tools. We present a formulation for the federated learning paradigm from a signal processing perspective, and survey a set of candidate approaches for tackling its unique challenges. We further provide guidelines for the design and adaptation of signal processing and communication methods to facilitate federated learning at large scale.

[1]  Xiang Li,et al.  On the Convergence of FedAvg on Non-IID Data , 2019, ICLR.

[2]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning , 2019, ICML.

[3]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[4]  H. Vincent Poor,et al.  Federated Learning With Differential Privacy: Algorithms and Performance Analysis , 2019, IEEE Transactions on Information Forensics and Security.

[5]  Li Li,et al.  A review of applications in federated learning , 2020, Comput. Ind. Eng..

[6]  Yonina C. Eldar,et al.  Over-the-Air Federated Learning From Heterogeneous Data , 2020, IEEE Transactions on Signal Processing.

[7]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2021, Found. Trends Mach. Learn..

[8]  Indranil Gupta,et al.  Generalized Byzantine-tolerant SGD , 2018, ArXiv.

[9]  Slawomir Stanczak,et al.  Nomographic Functions: Efficient Computation in Clustered Gaussian Sensor Networks , 2013, IEEE Transactions on Wireless Communications.

[10]  Slawomir Stanczak,et al.  Robust Analog Function Computation via Wireless Multiple-Access Channels , 2012, IEEE Transactions on Communications.

[11]  Caroline Fontaine,et al.  A Survey of Homomorphic Encryption for Nonspecialists , 2007, EURASIP J. Inf. Secur..

[12]  Deniz Gündüz,et al.  Federated Learning Over Wireless Fading Channels , 2019, IEEE Transactions on Wireless Communications.

[13]  Yonina C. Eldar,et al.  Communication-efficient federated learning , 2021, Proceedings of the National Academy of Sciences.

[14]  Shiho Moriai,et al.  Privacy-Preserving Deep Learning via Additively Homomorphic Encryption , 2018, IEEE Transactions on Information Forensics and Security.

[15]  Omid Salehi-Abari,et al.  AirShare: Distributed coherent transmission made seamless , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[16]  Xukan Ran,et al.  Deep Learning With Edge Computing: A Review , 2019, Proceedings of the IEEE.

[17]  Wojciech Samek,et al.  Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[19]  Tony Q. S. Quek,et al.  Multi-Armed Bandit-Based Client Scheduling for Federated Learning , 2020, IEEE Transactions on Wireless Communications.

[20]  Deniz Gündüz,et al.  Blind Federated Edge Learning , 2020, IEEE Transactions on Wireless Communications.

[21]  Zhi Ding,et al.  Federated Learning via Over-the-Air Computation , 2018, IEEE Transactions on Wireless Communications.

[22]  Yonina C. Eldar,et al.  The Communication-Aware Clustered Federated Learning Problem , 2020, 2020 IEEE International Symposium on Information Theory (ISIT).

[23]  H. Vincent Poor,et al.  Energy-Efficient Resource Allocation in Wireless Networks , 2007, IEEE Signal Processing Magazine.

[24]  Song Han,et al.  Deep Leakage from Gradients , 2019, NeurIPS.

[25]  Canh Dinh,et al.  Federated Learning Over Wireless Networks: Convergence Analysis and Resource Allocation , 2019, IEEE/ACM Transactions on Networking.

[26]  BOUZIANE BRIK,et al.  Federated Learning for UAVs-Enabled Wireless Networks: Use Cases, Challenges, and Open Problems , 2020, IEEE Access.

[27]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[28]  Mehryar Mohri,et al.  Agnostic Federated Learning , 2019, ICML.

[29]  Lingjia Liu,et al.  Federated Dynamic Spectrum Access , 2021, ArXiv.

[30]  Jie Xu,et al.  Federated Learning for Healthcare Informatics , 2019, ArXiv.

[31]  Zixiang Xiong,et al.  Distributed source coding for sensor networks , 2004, IEEE Signal Processing Magazine.

[32]  Kannan Ramchandran,et al.  Robust Federated Learning in a Heterogeneous Environment , 2019, ArXiv.

[33]  Lili Su,et al.  Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent , 2019, PERV.

[34]  Soo-Hyun Park,et al.  Multiagent DDPG-Based Deep Learning for Smart Ocean Federated Learning IoT Networks , 2020, IEEE Internet of Things Journal.

[35]  H. Vincent Poor,et al.  Scheduling Policies for Federated Learning in Wireless Networks , 2019, IEEE Transactions on Communications.

[36]  Brian M. Sadler,et al.  A Survey of Dynamic Spectrum Access , 2007, IEEE Signal Processing Magazine.

[37]  Dan Alistarh,et al.  QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks , 2016, 1610.02132.

[38]  Yonina C. Eldar,et al.  UVeQFed: Universal Vector Quantization for Federated Learning , 2021, IEEE Transactions on Signal Processing.

[39]  Yonina C. Eldar,et al.  Bayesian Federated Learning over Wireless Networks , 2020, ArXiv.

[40]  Deniz Gündüz,et al.  Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[41]  Hubert Eichner,et al.  Federated Learning for Mobile Keyboard Prediction , 2018, ArXiv.

[42]  Kobi Cohen,et al.  On Analog Gradient Descent Learning Over Multiple Access Fading Channels , 2019, IEEE Transactions on Signal Processing.

[43]  Rui Zhang,et al.  A Hybrid Approach to Privacy-Preserving Federated Learning , 2018, Informatik Spektrum.

[44]  Yonina C. Eldar,et al.  Collaborative Inference via Ensembles on the Edge , 2021, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[45]  Kannan Ramchandran,et al.  Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates , 2018, ICML.

[46]  Walter T. Federer,et al.  Sequential Design of Experiments , 1967 .

[47]  Osvaldo Simeone,et al.  Privacy for Free: Wireless Federated Learning via Uncoded Transmission With Adaptive Power Control , 2020, IEEE Journal on Selected Areas in Communications.

[48]  Y. Mansour,et al.  Three Approaches for Personalization with Applications to Federated Learning , 2020, ArXiv.

[49]  Yishay Mansour,et al.  Multiple Source Adaptation and the Rényi Divergence , 2009, UAI.

[50]  Kaibin Huang,et al.  Broadband Analog Aggregation for Low-Latency Federated Edge Learning , 2018, IEEE Transactions on Wireless Communications.

[51]  Cong Shen,et al.  Design and Analysis of Uplink and Downlink Communications for Federated Learning , 2020, IEEE Journal on Selected Areas in Communications.

[52]  Shiva Raj Pokhrel,et al.  A Blockchained Federated Learning Framework for Cognitive Computing in Industry 4.0 Networks , 2021, IEEE Transactions on Industrial Informatics.

[53]  H. Vincent Poor,et al.  Power Allocation for Artificial-Noise Secure MIMO Precoding Systems , 2014, IEEE Transactions on Signal Processing.

[54]  William J. Dally,et al.  Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training , 2017, ICLR.

[55]  Jinyuan Jia,et al.  Local Model Poisoning Attacks to Byzantine-Robust Federated Learning , 2019, USENIX Security Symposium.

[56]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[57]  Guozhong An,et al.  The Effects of Adding Noise During Backpropagation Training on a Generalization Performance , 1996, Neural Computation.

[58]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[59]  Martin Jaggi,et al.  Sparsified SGD with Memory , 2018, NeurIPS.

[60]  Walid Saad,et al.  Wireless Communications for Collaborative Federated Learning , 2020, IEEE Communications Magazine.

[61]  Walid Saad,et al.  A Joint Learning and Communications Framework for Federated Learning Over Wireless Networks , 2021, IEEE Transactions on Wireless Communications.

[62]  Ariela Zeira,et al.  Resource Management and Model Personalization for Federated Learning over Wireless Edge Networks , 2021, J. Sens. Actuator Networks.

[63]  Tomer Gafni,et al.  Distributed Learning over Markovian Fading Channels for Stable Spectrum Access , 2021, IEEE Access.

[64]  Sarvar Patel,et al.  Practical Secure Aggregation for Privacy-Preserving Machine Learning , 2017, IACR Cryptol. ePrint Arch..

[65]  Kamyar Azizzadenesheli,et al.  signSGD: compressed optimisation for non-convex problems , 2018, ICML.

[66]  Michael Gastpar,et al.  Computation Over Multiple-Access Channels , 2007, IEEE Transactions on Information Theory.

[67]  R. Gray,et al.  Dithered Quantizers , 1993, Proceedings. 1991 IEEE International Symposium on Information Theory.

[68]  Branka Vucetic,et al.  Over-the-Air Computation Systems: Optimization, Analysis and Scaling Laws , 2020, IEEE Transactions on Wireless Communications.