A Discrete-Time Multi-Hop Consensus Protocol for Decentralized Federated Learning

This paper presents a Federated Learning (FL) algorithm that allows the decentralization of all FL solutions that employ a model-averaging procedure. The proposed algorithm proves to be capable of attaining faster convergence rates and no performance loss against the starting centralized FL implementation with a reduced communication overhead compared to existing consensus-based and centralized solutions. To this end, a Multi-Hop consensus protocol, originally presented in the scope of dynamical system consensus theory, leveraging on standard Lyapunov stability discussions, has been proposed to assure that all federation clients share the same average model employing only information obtained from their ${m}$ -step neighbours. Experimental results on different communication topologies and the MNIST and MedMNIST v2 datasets validate the algorithm properties demonstrating a performance drop, compared with centralized FL setting, of about 1%.

[1]  T. Spyropoulos,et al.  FedDec: Peer-to-peer Aided Federated Learning , 2023, ArXiv.

[2]  Feifei Chen,et al.  FlexiFed: Personalized Federated Learning for Edge Clients with Heterogeneous Model Architectures , 2023, WWW.

[3]  Dun Zeng,et al.  A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness and Privacy , 2023, WWW.

[4]  Yi Shi,et al.  Improving the Model Consistency of Decentralized Federated Learning , 2023, ICML.

[5]  Junxiu Liu,et al.  Federated learning-based vertebral body segmentation , 2022, Eng. Appl. Artif. Intell..

[6]  O. Simeone,et al.  Channel-Driven Decentralized Bayesian Federated Learning for Trustworthy Decision Making in D2D Networks , 2022, ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[7]  Mi Zhang,et al.  PyramidFL: a fine-grained client selection framework for efficient federated learning , 2022, MobiCom.

[8]  Gautam Srivastava,et al.  Blockchain-based federated learning with checksums to increase security in Internet of Things solutions , 2022, Journal of Ambient Intelligence and Humanized Computing.

[9]  Md Zakirul Alam Bhuiyan,et al.  PSDF: Privacy-aware IoV Service Deployment with Federated Learning in Cloud-Edge Computing , 2022, ACM Trans. Intell. Syst. Technol..

[10]  S. Manfredi,et al.  A Weighted Average Consensus Approach for Decentralized Federated Learning , 2022, Machine Intelligence Research.

[11]  Cho-Jui Hsieh,et al.  FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  P. Li,et al.  On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond , 2022, NeurIPS.

[13]  Qing Ling,et al.  Confederated Learning: Federated Learning With Decentralized Edge Servers , 2022, IEEE Transactions on Signal Processing.

[14]  Jing Jiang,et al.  Personalized Federated Learning With Graph , 2022, 2203.00829.

[15]  Jinkang Zhu,et al.  DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology , 2021, ArXiv.

[16]  Bingbing Ni,et al.  MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification , 2021, Scientific Data.

[17]  Sebastian U. Stich,et al.  ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training , 2021, ICML.

[18]  Le Liang,et al.  Decentralized Federated Learning With Unreliable Communications , 2021, IEEE Journal of Selected Topics in Signal Processing.

[19]  Eryk Dutkiewicz,et al.  Federated Learning Framework With Straggling Mitigation and Privacy-Awareness for AI-Based Mobile Application Services , 2021, IEEE Transactions on Mobile Computing.

[20]  Qiang Yang,et al.  Towards Personalized Federated Learning , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Josep Domingo-Ferrer,et al.  Achieving Security and Privacy in Federated Learning Systems: Survey, Research Challenges and Future Directions , 2020, Eng. Appl. Artif. Intell..

[22]  Andrea Acevedo,et al.  A dataset of microscopic peripheral blood cell images for development of automatic recognition systems , 2020, Data in brief.

[23]  Monica Nicoli,et al.  Federated Learning With Cooperating Devices: A Consensus Approach for Massive IoT Networks , 2019, IEEE Internet of Things Journal.

[24]  Bingsheng He,et al.  A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection , 2019, IEEE Transactions on Knowledge and Data Engineering.

[25]  Anit Kumar Sahu,et al.  Federated Optimization in Heterogeneous Networks , 2018, MLSys.

[26]  Javad Khazaei,et al.  Consensus Control for Energy Storage Systems , 2018, IEEE Transactions on Smart Grid.

[27]  Abien Fred Agarap.  Deep Learning using Rectified Linear Units (ReLU) , 2018, ArXiv.

[28]  Zongli Lin,et al.  Global leader-following consensus of a group of general linear systems using bounded controls , 2016, Autom..

[29]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[30]  Chia-Chi Chu,et al.  Consensus-based droop control synthesis for multiple DICs in isolated micro-grids , 2015, 2015 IEEE Power & Energy Society General Meeting.

[31]  Sabato Manfredi,et al.  A theoretical analysis of multi-hop consensus algorithms for wireless networks: Trade off among reliability, responsiveness and delay tolerance , 2014, Ad Hoc Networks.

[32]  Sabato Manfredi,et al.  Design of a multi-hop dynamic consensus algorithm over wireless sensor networks , 2013 .

[33]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[34]  Sonia Martínez,et al.  Discrete-time dynamic average consensus , 2010, Autom..

[35]  Kim D. Listmann,et al.  Consensus for formation control of nonholonomic mobile robots , 2009, 2009 IEEE International Conference on Robotics and Automation.

[36]  Reza Olfati-Saber,et al.  Consensus and Cooperation in Networked Multi-Agent Systems , 2007, Proceedings of the IEEE.

[37]  R.M. Murray,et al.  Multi-Hop Relay Protocols for Fast Consensus Seeking , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[38]  Online Services.  About The Department , 2006 .

[39]  Richard M. Murray,et al.  Information flow and cooperative control of vehicle formations , 2004, IEEE Transactions on Automatic Control.

[40]  Richard M. Murray,et al.  Consensus problems in networks of agents with switching topology and time-delays , 2004, IEEE Transactions on Automatic Control.

[41]  Asrar U. H. Sheikh,et al.  Wireless Communications , 2003, Springer US.

[42]  R. Murray,et al.  Consensus protocols for networks of dynamic agents , 2003, Proceedings of the 2003 American Control Conference, 2003..

[43]  Naomi Ehrich Leonard,et al.  Virtual leaders, artificial potentials and coordinated control of groups , 2001, Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228).

[44]  S C Weller,et al.  Assessing Rater Performance without a "Gold Standard" Using Consensus Theory , 1997, Medical decision making : an international journal of the Society for Medical Decision Making.

[45]  Jon Atli Benediktsson,et al.  Consensus theoretic classification methods , 1992, IEEE Trans. Syst. Man Cybern..

[46]  John N. Tsitsiklis,et al.  Problems in decentralized decision making and computation , 1984 .

[47]  John N. Tsitsiklis,et al.  Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[48]  V. Borkar,et al.  Asymptotic agreement in distributed estimation , 1982 .

[49]  M. Degroot.  Reaching a Consensus , 1974 .

[50]  Weisong Shi,et al.  VCD-FL: Verifiable, Collusion-Resistant, and Dynamic Federated Learning , 2023, IEEE Transactions on Information Forensics and Security.

[51]  Jinyang Guo,et al.  Visual Object Detection for Privacy-Preserving Federated Learning , 2023, IEEE Access.

[52]  Philip H. S. Torr,et al.  FedSR: A Simple and Effective Domain Generalization Method for Federated Learning , 2022, NeurIPS.

[53]  Carlos Carrascosa,et al.  Convergence of Weighted-average consensus for undirected graphs , 2014 .

[54]  Richard M. Murray,et al.  DISTRIBUTED COOPERATIVE CONTROL OF MULTIPLE VEHICLE FORMATIONS USING STRUCTURAL POTENTIAL FUNCTIONS , 2002 .

[55]  J. A. Fax,et al.  Graph Laplacians and Stabilization of Vehicle Formations , 2002 .

[56]  J. A. Fax.  Optimal and Cooperative Control of Vehicle Formations , 2002 .

[57]  Seif Haridi,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[58]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[59]  Katsuhiko Ogata,et al.  Discrete-time control systems , 1987 .

[60]  Chuhan Wu,et al.  Communication-efficient federated learning via knowledge distillation , 2021, Nature Communications.