Next Generation Federated Learning for Edge Devices: An Overview

Federated learning (FL) is a popular distributed machine learning paradigm involving numerous edge devices with enhanced privacy protection. Recently, an extensive literature has been developing on the research which aims at promoting the innovations of FL. Motivated by the explosive growth in FL research, this paper studies the next generation of Federated Learning for edge devices. We identify two key challenges, system efficiency and data heterogeneity, which impede the development of FL. We introduce some representative works which contribute to these challenges. Besides, we anticipate the future directions of FL for edge devices and provide guidance for future FL research.

[1]  Qian Yang,et al.  A Survey on Heterogeneous Federated Learning , 2022, ArXiv.

[2]  Yiran Chen,et al.  Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction , 2022, ArXiv.

[3]  Aryan Mokhtari,et al.  Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity , 2020, IEEE Journal on Selected Areas in Information Theory.

[4]  K. Ramchandran,et al.  An Efficient Framework for Clustered Federated Learning , 2020, IEEE Transactions on Information Theory.

[5]  H. Li,et al.  LotteryFL: Empower Edge Intelligence with Personalized and Communication-Efficient Federated Learning , 2021, IFIP International Information Security Conference.

[6]  Hai Li,et al.  FedMask: Joint Computation and Communication-Efficient Personalized Federated Learning via Heterogeneous Masking , 2021, SenSys.

[7]  H. Li,et al.  Hermes: an efficient federated learning framework for heterogeneous mobile clients , 2021, MobiCom.

[8]  Lawrence Carin,et al.  FLOP: Federated Learning on Medical Datasets using Partial Networks , 2021, KDD.

[9]  Taiji Suzuki,et al.  Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning , 2021, ICML.

[10]  Hongbin Zhu,et al.  Federated Learning with Class Imbalance Reduction , 2020, 2021 29th European Signal Processing Conference (EUSIPCO).

[11]  Albert Y. Zomaya,et al.  An Efficiency-Boosting Client Selection Scheme for Federated Learning With Fairness Guarantee , 2020, IEEE Transactions on Parallel and Distributed Systems.

[12]  M. Chowdhury,et al.  Oort: Efficient Federated Learning via Guided Participant Selection , 2020, OSDI.

[13]  J. Bilmes,et al.  Diverse Client Selection for Federated Learning:Submodularity and Convergence Analysis , 2021 .

[14]  Qinghua Liu,et al.  Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization , 2020, NeurIPS.

[15]  T. Hsu,et al.  Federated Visual Classification with Real-World Data Distribution , 2020, ECCV.

[16]  Ming Li,et al.  Wireless Federated Learning with Local Differential Privacy , 2020, 2020 IEEE International Symposium on Information Theory (ISIT).

[17]  Ruslan Salakhutdinov,et al.  Think Locally, Act Globally: Federated Learning with Local and Global Representations , 2020, ArXiv.

[18]  Ahmed M. Abdelmoniem,et al.  On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning , 2019, AAAI.

[19]  H. Vincent Poor,et al.  Federated Learning With Differential Privacy: Algorithms and Performance Analysis , 2019, IEEE Transactions on Information Forensics and Security.

[20]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning , 2019, ICML.

[21]  Anit Kumar Sahu,et al.  Federated Optimization in Heterogeneous Networks , 2018, MLSys.

[22]  Aryan Mokhtari,et al.  Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach , 2020, NeurIPS.

[23]  Ilana Segall,et al.  Federated Learning for Ranking Browser History Suggestions , 2019, ArXiv.

[24]  Hubert Eichner,et al.  Federated Evaluation of On-device Personalization , 2019, ArXiv.

[25]  Nadav Israel,et al.  Overcoming Forgetting in Federated Learning on Non-IID Data , 2019, ArXiv.

[26]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning , 2019, ArXiv.

[27]  Enhong Chen,et al.  Variance Reduced Local SGD with Lower Communication Complexity , 2019, ArXiv.

[28]  Jakub Konecný,et al.  Improving Federated Learning Personalization via Model Agnostic Meta Learning , 2019, ArXiv.

[29]  Swaroop Ramaswamy,et al.  Federated Learning for Emoji Prediction in a Mobile Keyboard , 2019, ArXiv.

[30]  Song Han,et al.  Deep Leakage from Gradients , 2019, NeurIPS.

[31]  Vladimir Braverman,et al.  Communication-efficient distributed SGD with Sketching , 2019, NeurIPS.

[32]  Takayuki Nishio,et al.  Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge , 2018, ICC 2019 - 2019 IEEE International Conference on Communications (ICC).

[33]  Michael Carbin,et al.  The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks , 2018, ICLR.

[34]  Anit Kumar Sahu,et al.  On the Convergence of Federated Optimization in Heterogeneous Networks , 2018, ArXiv.

[35]  Hubert Eichner,et al.  APPLIED FEDERATED LEARNING: IMPROVING GOOGLE KEYBOARD QUERY SUGGESTIONS , 2018, ArXiv.

[36]  Mehdi Bennis,et al.  Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data , 2018, ArXiv.

[37]  Hubert Eichner,et al.  Federated Learning for Mobile Keyboard Prediction , 2018, ArXiv.

[38]  S. V. Lier.  Robustness of Federated Averaging for Non-IID Data , 2018 .

[39]  A. Arkader,et al.  Hermes , 2020, Pediatric Radiology.

[40]  Cong Xu,et al.  TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning , 2017, NIPS.

[41]  Kenneth Heafield,et al.  Sparse Communication for Distributed Gradient Descent , 2017, EMNLP.

[42]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[43]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[44]  Paul Voigt,et al.  The EU General Data Protection Regulation (GDPR) , 2017 .

[45]  Jian Cheng,et al.  Quantized Convolutional Neural Networks for Mobile Devices , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Somesh Jha,et al.  Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures , 2015, CCS.

[47]  Pritish Narayanan,et al.  Deep Learning with Limited Numerical Precision , 2015, ICML.

[48]  Ming Yang,et al.  Compressing Deep Convolutional Networks using Vector Quantization , 2014, ArXiv.

[49]  Dong Yu,et al.  1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs , 2014, INTERSPEECH.

[50]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.