Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data

Learning-based applications have demonstrated practical use cases in ubiquitous environments and amplified interest in exploiting the data stored on users' mobile devices. Distributed learning algorithms aim to leverage such distributed and diverse data to learn a global phenomena by performing training amongst participating devices and repeatedly aggregating their local models' parameters into a global model. Federated learning is a promising paradigm that allows for extending local training among the participant devices before aggregating the parameters, offering better communication efficiency. However, in the cases where the participants' data are strongly skewed (i.e., non-IID), the model accuracy can significantly drop. To face this challenge, we leverage the edge computing paradigm to design a hierarchical learning system that performs Federated Gradient Descent on the user-edge layer and Federated Averaging on the edge-cloud layer. In this hierarchical architecture, the users are assigned to different edges, such that edge-level data distributions turn to be close to IID. We formalize and optimize this user-edge assignment problem to minimize classes' distribution distance between edge nodes, which enhances the Federated Averaging performance. Our experiments on multiple real-world datasets show that the proposed optimized assignment is tractable and leads to faster convergence of models towards a better accuracy value.

[1]  Deeparnab Chakrabarty,et al.  Knapsack Problems , 2008 .

[2]  Christian Kirches,et al.  Mixed-integer nonlinear optimization*† , 2013, Acta Numerica.

[3]  Mykel J. Kochenderfer,et al.  Algorithms for Optimization , 2019 .

[4]  Xu Chen,et al.  In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning , 2018, IEEE Network.

[5]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[6]  Mugen Peng,et al.  Joint Optimization of Data Sampling and User Selection for Federated Learning in the Mobile Edge Computing Systems , 2020, 2020 IEEE International Conference on Communications Workshops (ICC Workshops).

[7]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[8]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[9]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[10]  Mehdi Bennis,et al.  Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data , 2018, ArXiv.

[11]  Jun Zhang,et al.  Edge-Assisted Hierarchical Federated Learning with Non-IID Data , 2019, ArXiv.

[12]  Ghaith Hattab,et al.  Reconfigurable Wireless Networks , 2014, Proceedings of the IEEE.

[13]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[14]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[15]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[16]  Ligang He,et al.  Accelerating Federated Learning over Reliability-Agnostic Clients in Mobile Edge Computing Systems , 2020, ArXiv.

[17]  Sarvar Patel,et al.  Practical Secure Aggregation for Federated Learning on User-Held Data , 2016, ArXiv.

[18]  Yuanyuan Yang,et al.  Towards Efficient Scheduling of Federated Mobile Devices Under Computational and Statistical Heterogeneity , 2020, IEEE Transactions on Parallel and Distributed Systems.

[19]  K. B. Letaief,et al.  A Survey on Mobile Edge Computing: The Communication Perspective , 2017, IEEE Communications Surveys & Tutorials.

[20]  Yue Zhao,et al.  Federated Learning with Non-IID Data , 2018, ArXiv.

[21]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[22]  Hao Wang,et al.  Optimizing Federated Learning on Non-IID Data with Reinforcement Learning , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[23]  Rong-Rong Chen,et al.  Local Averaging Helps: Hierarchical Federated Learning and Convergence Analysis , 2020, ArXiv.

[24]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[26]  Xuanzhe Liu,et al.  Hierarchical Federated Learning through LAN-WAN Orchestration , 2020, ArXiv.

[27]  Ying-Chang Liang,et al.  Federated Learning in Mobile Edge Networks: A Comprehensive Survey , 2020, IEEE Communications Surveys & Tutorials.

[28]  Takayuki Nishio,et al.  Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge , 2018, ICC 2019 - 2019 IEEE International Conference on Communications (ICC).

[29]  Albert Y. Zomaya,et al.  Federated Learning over Wireless Networks: Optimization Model Design and Analysis , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[30]  Samy Bengio,et al.  Revisiting Distributed Synchronous SGD , 2016, ArXiv.

[31]  Tzu-Ming Harry Hsu,et al.  Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification , 2019, ArXiv.

[32]  Kin K. Leung,et al.  Adaptive Federated Learning in Resource Constrained Edge Computing Systems , 2018, IEEE Journal on Selected Areas in Communications.

[33]  Liang Liang,et al.  Self-Balancing Federated Learning With Global Imbalanced Data in Mobile Systems , 2021, IEEE Transactions on Parallel and Distributed Systems.

[34]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[35]  Klaus-Robert Müller,et al.  Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[36]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.