Personalized Decentralized Federated Learning with Knowledge Distillation

Personalization in federated learning (FL) functions as a coordinator for clients with high variance in data or behavior. Ensuring the convergence of these clients' models relies on how closely users collaborate with those with similar patterns or preferences. However, it is generally challenging to quantify similarity under limited knowledge about other users' models given to users in a decentralized network. To cope with this issue, we propose a personalized and fully decentralized FL algorithm, leveraging knowledge distillation techniques to empower each device so as to discern statistical distances between local models. Each client device can enhance its performance without sharing local data by estimating the similarity between two intermediate outputs from feeding local samples as in knowledge distillation. Our empirical studies demonstrate that the proposed algorithm improves the test accuracy of clients in fewer iterations under highly non-independent and identically distributed (non-i.i.d.) data distributions and is beneficial to agents with small datasets, even without the need for a central server.

[1]  Zachary B. Charles,et al.  Motley: Benchmarking Heterogeneity and Personalization in Federated Learning , 2022, ArXiv.

[2]  Tianyi Zhou,et al.  Learning to Collaborate in Decentralized Learning of Personalized Models , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  G. Neglia,et al.  Personalized Federated Learning through Local Memorization , 2021, ICML.

[4]  A. Gasnikov,et al.  Decentralized personalized federated learning: Lower bounds and optimal algorithm for all personalization modes , 2021, EURO J. Comput. Optim..

[5]  Urmish Thakker,et al.  A Survey on Federated Learning for Resource-Constrained IoT Devices , 2021, IEEE Internet of Things Journal.

[6]  A. Bellet,et al.  Refined Convergence and Topology Learning for Decentralized Optimization with Heterogeneous Data , 2022 .

[7]  Tong Chen,et al.  A Decentralized Collaborative Learning Framework Across Heterogeneous Devices for Personalized Predictive Analytics , 2022, ArXiv.

[8]  Giovanni Neglia,et al.  Federated Multi-Task Learning under a Mixture of Distributions , 2021, NeurIPS.

[9]  A. Gasnikov,et al.  Decentralized Personalized Federated Min-Max Problems , 2021, ArXiv.

[10]  Siddharth Divi,et al.  Unifying Distillation with Personalization in Federated Learning , 2021, ArXiv.

[11]  Lingyang Chu,et al.  Personalized Federated Learning: An Attentive Collaboration Approach , 2020, ArXiv.

[12]  Michael Moeller,et al.  Inverting Gradients - How easy is it to break privacy in federated learning? , 2020, NeurIPS.

[13]  Y. Mansour,et al.  Three Approaches for Personalization with Applications to Federated Learning , 2020, ArXiv.

[14]  Aryan Mokhtari,et al.  Personalized Federated Learning: A Meta-Learning Approach , 2020, ArXiv.

[15]  Yasaman Khazaeni,et al.  Federated Learning with Matched Averaging , 2020, ICLR.

[16]  Solmaz Niknam,et al.  Federated Learning for Wireless Communications: Motivation, Opportunities, and Challenges , 2019, IEEE Communications Magazine.

[17]  Xiang Li,et al.  On the Convergence of FedAvg on Non-IID Data , 2019, ICLR.

[18]  Klaus-Robert Müller,et al.  Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Marc Tommasi,et al.  Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs , 2019, AISTATS.

[20]  Jakub Konecný,et al.  Improving Federated Learning Personalization via Model Agnostic Meta Learning , 2019, ArXiv.

[21]  Mehdi Bennis,et al.  Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data , 2018, ArXiv.

[22]  Joshua Achiam,et al.  On First-Order Meta-Learning Algorithms , 2018, ArXiv.

[23]  Geoffrey E. Hinton,et al.  Large scale distributed neural network training through online distillation , 2018, ICLR.

[24]  Rachid Guerraoui,et al.  Personalized and Private Peer-to-Peer Machine Learning , 2017, AISTATS.

[25]  Ameet Talwalkar,et al.  Federated Multi-Task Learning , 2017, NIPS.

[26]  Marc Tommasi,et al.  Decentralized Collaborative Learning of Personalized Models over Networks , 2016, AISTATS.

[27]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[28]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[29]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.