Precision-Weighted Federated Learning

Federated Learning using the Federated Averaging algorithm has shown great advantages for large-scale applications that rely on collaborative learning, especially when the training data is either unbalanced or inaccessible due to privacy constraints. We hypothesize that Federated Averaging underestimates the full extent of heterogeneity of data when the aggregation is performed. We propose Precision-weighted Federated Learning1 a novel algorithm that takes into account the second raw moment (uncentered variance) of the stochastic gradient when computing the weighted average of the parameters of independent models trained in a Federated Learning setting. With Precision-weighted Federated Learning, we address the communication and statistical challenges for the training of distributed models with private data and provide an alternate averaging scheme that leverages the heterogeneity of the data when it has a large diversity of features in its composition. Our method was evaluated using three standard image classification datasets (MNIST, Fashion-MNIST, and CIFAR) with two different data partitioning strategies (independent and identically distributed (IID), and non-identical and non-independent (non-IID)) to measure the performance and speed of our method in resource-constrained environments, such as mobile and IoT devices. The experimental results demonstrate that we can obtain a good balance between computational efficiency and convergence rates with Precision-weighted Federated Learning. Our performance evaluations show 9% better predictions with MNIST, 18% with Fashion-MNIST, and 5% with CIFAR-10 in the non-IID setting. Further reliability evaluations ratify the stability in our method by reaching a 99% reliability index with IID partitions and 96% with non-IID partitions. In addition, we obtained a 20x speedup on Fashion-MNIST with only 10 clients and up to 37x with 100 clients participating in the aggregation concurrently per communication round. The results indicate that Precision-weighted Federated Learning is an effective and faster alternative approach for aggregating private data, especially in domains where data is highly heterogeneous.

[1]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[2]  H. Vincent Poor,et al.  Federated Learning With Differential Privacy: Algorithms and Performance Analysis , 2019, IEEE Transactions on Information Forensics and Security.

[3]  Bruce R. Rosen,et al.  Distributed deep learning networks among institutions for medical imaging , 2018, J. Am. Medical Informatics Assoc..

[4]  Luca Benini,et al.  Efficient image dataset classification difficulty estimation for predicting deep-learning accuracy , 2018, The Visual Computer.

[5]  Chuan Chen,et al.  Collaborative deep learning across multiple data centers , 2018, Science China Information Sciences.

[6]  Zhenguo Li,et al.  Federated Meta-Learning for Recommendation , 2018, ArXiv.

[7]  Hyo-Eun Kim,et al.  Keep and Learn: Continual Learning by Constraining the Latent Space for Knowledge Preservation in Neural Networks , 2018, MICCAI.

[8]  Yehuda Lindell,et al.  Privacy Preserving Data Mining , 2002, Journal of Cryptology.

[9]  Shinichi Nakagawa,et al.  Methodological issues and advances in biological meta-analysis , 2012, Evolutionary Ecology.

[10]  Sarvar Patel,et al.  Practical Secure Aggregation for Federated Learning on User-Held Data , 2016, ArXiv.

[11]  Yue Zhao,et al.  Federated Learning with Non-IID Data , 2018, ArXiv.

[12]  Ayman El-Baz,et al.  Accurate Diabetes Risk Stratification Using Machine Learning: Role of Missing Value and Outliers , 2018, Journal of Medical Systems.

[13]  Tassilo Klein,et al.  Differentially Private Federated Learning: A Client Level Perspective , 2017, ArXiv.

[14]  Ramesh Raskar,et al.  Split learning for health: Distributed deep learning without sharing raw patient data , 2018, ArXiv.

[15]  Vitaly Shmatikov,et al.  Exploiting Unintended Feature Leakage in Collaborative Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[16]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[17]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[18]  Hubert Eichner,et al.  APPLIED FEDERATED LEARNING: IMPROVING GOOGLE KEYBOARD QUERY SUGGESTIONS , 2018, ArXiv.

[19]  Rui Zhang,et al.  A Hybrid Approach to Privacy-Preserving Federated Learning , 2018, Informatik Spektrum.

[20]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[21]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[22]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[23]  Razvan Pascanu,et al.  Natural Gradient Revisited , 2013, ICLR.

[24]  Mathias Johanson,et al.  Big Automotive Data: Leveraging large volumes of data for knowledge-driven product development , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[25]  Tzu-Ming Harry Hsu,et al.  Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification , 2019, ArXiv.

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[28]  Taghi M. Khoshgoftaar,et al.  Deep learning applications and challenges in big data analytics , 2015, Journal of Big Data.

[29]  Ameet Talwalkar,et al.  Federated Multi-Task Learning , 2017, NIPS.

[30]  Evangelos Evangelou,et al.  Heterogeneity in Meta-Analyses of Genome-Wide Association Investigations , 2007, PloS one.

[31]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[32]  Anusha Lalitha,et al.  Fully Decentralized Federated Learning , 2018 .

[33]  Chunxiao Jiang,et al.  Information Security in Big Data: Privacy and Data Mining , 2014, IEEE Access.

[34]  D Y Lin,et al.  Meta‐analysis of genome‐wide association studies: no efficiency gain in using individual participant data , 2009, Genetic epidemiology.

[35]  Xiaojiang Du,et al.  A Distributed Deep Learning System for Web Attack Detection on Edge Devices , 2020, IEEE Transactions on Industrial Informatics.

[36]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[37]  Richard E. Turner,et al.  Partitioned Variational Inference: A unified framework encompassing federated and continual learning , 2018, ArXiv.

[38]  Elisa Bertino,et al.  State-of-the-art in privacy preserving data mining , 2004, SGMD.

[39]  Susan Michie,et al.  Statistical methodologies to pool across multiple intervention studies , 2016, Translational behavioral medicine.