Sparse Personalized Federated Learning via Maximizing Correlation

Federated Learning (FL) is a collaborative machine learning technique to train a global model without obtaining clients’ private data. The main challenges in FL are statistical diversity among clients, limited computing capability among client equipments and the excessive communication overhead between server and clients. To address these challenges, we propose a novel sparse personalized federated learning scheme via maximizing correlation (FedMac). By incorporating an approximated `1-norm and the correlation between client models and global model into standard FL loss function, the performance on statistical diversity data is improved and the communicational and computational loads required in the network are reduced compared with non-sparse FL. Convergence analysis shows that the sparse constraints in FedMac do not affect the convergence rate of the global model, and theoretical results show that FedMac can achieve good sparse personalization, which is better than the personalized methods based on `2-norm. Experimentally, we demonstrate the benefits of this sparse personalization architecture compared with the state-of-the-art personalization methods (e.g. FedMac respectively achieves 98.95%, 99.37%, 90.90% and 89.06% accuracy on the MNIST, FMNIST, CIFAR-100 and Synthetic datasets under non-i.i.d. variants).

[1]  Jian Pei,et al.  Personalized Cross-Silo Federated Learning on Non-IID Data , 2020, AAAI.

[2]  Spyridon Bakas,et al.  Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data , 2020, Scientific Reports.

[3]  Shusen Wang,et al.  Communication-Efficient Local Decentralized SGD Methods , 2019 .

[4]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[5]  Klaus-Robert Müller,et al.  Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[6]  YinchuanLi,et al.  Structured Directional Pruning via Perturbation Orthogonal Projection , 2021, ArXiv.

[7]  Ameet Talwalkar,et al.  Federated Multi-Task Learning , 2017, NIPS.

[8]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning , 2019, ICML.

[9]  Zhi Zhang,et al.  Bag of Tricks for Image Classification with Convolutional Neural Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[11]  Rong Jin,et al.  On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization , 2019, ICML.

[12]  Sunav Choudhary,et al.  Federated Learning with Personalization Layers , 2019, ArXiv.

[13]  Ming Liu,et al.  Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems , 2019, IEEE Robotics and Automation Letters.

[14]  Wei Cui,et al.  Compressed sensing with prior information via maximizing correlation , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[15]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[16]  Giuseppe Ateniese,et al.  Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning , 2017, CCS.

[17]  Miguel R. D. Rodrigues,et al.  Compressed Sensing with Prior Information: Optimal Strategies, Geometry, and Bounds , 2014, ArXiv.

[18]  Yulong Liu,et al.  Stable Recovery of Structured Signals From Corrupted Sub-Gaussian Measurements , 2019, IEEE Transactions on Information Theory.

[19]  Xu Zhang,et al.  Recovery of Structured Signals With Prior Information via Maximizing Correlation , 2017, IEEE Transactions on Signal Processing.

[20]  Abbas Mehrabian,et al.  A simple tool for bounding the deviation of random matrices on geometric sets , 2016, ArXiv.

[21]  Ali Dehghantanha,et al.  A survey on security and privacy of federated learning , 2021, Future Gener. Comput. Syst..

[22]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[23]  Yonina C. Eldar,et al.  UVeQFed: Universal Vector Quantization for Federated Learning , 2021, IEEE Transactions on Signal Processing.

[24]  Guang Cheng,et al.  Directional Pruning of Deep Neural Networks , 2020, NeurIPS.

[25]  Aryan Mokhtari,et al.  Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach , 2020, NeurIPS.

[26]  Mehrdad Mahdavi,et al.  Adaptive Personalized Federated Learning , 2020, ArXiv.

[27]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[28]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[29]  Micah J. Sheller,et al.  The future of digital health with federated learning , 2020, npj Digital Medicine.

[30]  John Wright,et al.  Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture , 2015, IEEE Transactions on Information Theory.

[31]  Alexander J. Smola,et al.  Parallelized Stochastic Gradient Descent , 2010, NIPS.

[32]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[33]  Nguyen H. Tran,et al.  Personalized Federated Learning with Moreau Envelopes , 2020, NeurIPS.

[34]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[35]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[36]  Xiaoyan Sun,et al.  Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Anit Kumar Sahu,et al.  Federated Optimization in Heterogeneous Networks , 2018, MLSys.

[38]  Hangyu Zhu,et al.  Federated Learning on Non-IID Data: A Survey , 2021, Neurocomputing.

[39]  T. Blumensath,et al.  Theory and Applications , 2011 .

[40]  R. Venkatesh Babu,et al.  Training Sparse Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).