Survey of Personalization Techniques for Federated Learning

Federated learning enables machine learning models to learn from private decentralized data without compromising privacy. The standard formulation of federated learning produces one shared model for all clients. Statistical heterogeneity due to non-IID distribution of data across devices often leads to scenarios where, for some clients, the local models trained solely on their private data perform better than the global shared model thus taking away their incentive to participate in the process. Several techniques have been proposed to personalize global models to work better for individual clients. This paper highlights the need for personalization and surveys recent research on this topic.

[1]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[2]  Paul Voigt,et al.  The EU General Data Protection Regulation (GDPR) , 2017 .

[3]  Sunav Choudhary,et al.  Federated Learning with Personalization Layers , 2019, ArXiv.

[4]  J. Schulman,et al.  Reptile: a Scalable Metalearning Algorithm , 2018 .

[5]  Junpu Wang,et al.  FedMD: Heterogenous Federated Learning via Model Distillation , 2019, ArXiv.

[6]  Kin K. Leung,et al.  Data Selection for Federated Learning with Relevant and Irrelevant Data at Clients , 2020, ArXiv.

[7]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[8]  Aryan Mokhtari,et al.  Personalized Federated Learning: A Meta-Learning Approach , 2020, ArXiv.

[9]  Hubert Eichner,et al.  Federated Evaluation of On-device Personalization , 2019, ArXiv.

[10]  Qiong Wu,et al.  Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge Based Framework , 2020, IEEE Open Journal of the Computer Society.

[11]  Kin K. Leung,et al.  Overcoming Noisy and Irrelevant Data in Federated Learning , 2020, 2020 25th International Conference on Pattern Recognition (ICPR).

[12]  Maria-Florina Balcan,et al.  Adaptive Gradient-Based Meta-Learning Methods , 2019, NeurIPS.

[13]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[14]  Johannes Schneider,et al.  Mass Personalization of Deep Learning , 2019, ArXiv.

[15]  Hubert Eichner,et al.  Federated Learning for Mobile Keyboard Prediction , 2018, ArXiv.

[16]  Joshua Achiam,et al.  On First-Order Meta-Learning Algorithms , 2018, ArXiv.

[17]  Zhenguo Li,et al.  Federated Meta-Learning for Recommendation , 2018, ArXiv.

[18]  Vitaly Shmatikov,et al.  Salvaging Federated Learning by Local Adaptation , 2020, ArXiv.

[19]  Vitaly Shmatikov,et al.  Exploiting Unintended Feature Leakage in Collaborative Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[20]  Sreeram Kannan,et al.  Improving Federated Learning Personalization via Model Agnostic Meta Learning , 2019, ArXiv.

[21]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[22]  Vitaly Shmatikov,et al.  Differential Privacy Has Disparate Impact on Model Accuracy , 2019, NeurIPS.

[23]  Lorien Y. Pratt,et al.  Discriminability-Based Transfer between Neural Networks , 1992, NIPS.

[24]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[25]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[26]  R. French.  Catastrophic forgetting in connectionist networks , 1999, Trends in Cognitive Sciences.

[27]  Yishay Mansour,et al.  Three Approaches for Personalization with Applications to Federated Learning , 2020, ArXiv.

[28]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[29]  Khe Chai Sim,et al.  An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models , 2019, INTERSPEECH.

[30]  G. Annas.  HIPAA regulations - a new era of medical-record privacy? , 2003, The New England journal of medicine.

[31]  Michail Vlachos,et al.  Personalization of Deep Learning , 2019, iDSC'19.

[32]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[33]  Fei Chen,et al.  Federated Meta-Learning with Fast Convergence and Efficient Communication , 2018 .

[34]  Ameet Talwalkar,et al.  Federated Multi-Task Learning , 2017, NIPS.

[35]  Filip Hanzely,et al.  Federated Learning of a Mixture of Global and Local Models , 2020, ArXiv.

[36]  H. Brendan McMahan,et al.  Learning Differentially Private Recurrent Language Models , 2017, ICLR.