Quantifying the Impact of Label Noise on Federated Learning

Federated Learning (FL) is a distributed machine learning paradigm where clients collaboratively train a model using their local (human-generated) datasets. While existing studies focus on FL algorithm development to tackle data heterogeneity across clients, the important issue of data quality (e.g., label noise) in FL is overlooked. This paper aims to fill this gap by providing a quantitative study on the impact of label noise on FL. We derive an upper bound for the generalization error that is linear in the clients' label noise level. Then we conduct experiments on MNIST and CIFAR-10 datasets using various FL algorithms. Our empirical results show that the global model accuracy linearly decreases as the noise level increases, which is consistent with our theoretical analysis. We further find that label noise slows down the convergence of FL training, and the global model tends to overfit when the noise level is high.

[1]  Qiang Yang,et al.  Towards Efficient Synchronous Federated Training: A Survey on System Optimization Strategies , 2021, IEEE Transactions on Big Data.

[2]  Bingsheng He,et al.  A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection , 2019, IEEE Transactions on Knowledge and Data Engineering.

[3]  Se-Young Yun,et al.  FedRN , 2022, Proceedings of the 31st ACM International Conference on Information & Knowledge Management.

[4]  N. Meratnia,et al.  Federated Learning with Noisy Labels , 2022, ArXiv.

[5]  S. Kambhampati.  Changing the nature of AI research , 2022, Commun. ACM.

[6]  Chuanyi Liu,et al.  Fed-DR-Filter: Using global data representation to reduce the impact of noisy labels on the performance of federated learning , 2022, Future Gener. Comput. Syst..

[7]  Xin Liu,et al.  Cross-Silo Federated Learning: Challenges and Opportunities , 2022, ArXiv.

[8]  Junyi Li,et al.  Communication-Efficient Robust Federated Learning with Noisy Labels , 2022, KDD.

[9]  X. Fang,et al.  Robust Federated Learning with Noisy and Heterogeneous Clients , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Phillip B. Gibbons,et al.  Federated Learning under Distributed Concept Drift , 2022, AISTATS.

[11]  S. Shakkottai,et al.  FedAvg with Fine Tuning: Local Updates Lead to Representation Learning , 2022, NeurIPS.

[12]  Tianyi Zhou,et al.  FedNoiL: A Simple Two-Level Sampling Method for Federated Learning with Noisy Labels , 2022, ArXiv.

[13]  S. Avestimehr,et al.  Federated Learning with Noisy User Feedback , 2022, NAACL.

[14]  Tony Q. S. Quek,et al.  FedCorr: Multi-Stage Federated Learning for Label Noise Correction , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  T. Khoshgoftaar,et al.  A Survey on Classifying Big Data with Label Noise , 2022, ACM J. Data Inf. Qual..

[16]  Xiaodong Yang,et al.  CLC: A Consensus-based Label Correction Approach in Federated Learning , 2022, ACM Trans. Intell. Syst. Technol..

[17]  Chao Huang,et al.  Incentivizing Data Contribution in Cross-Silo Federated Learning , 2022, ArXiv.

[18]  Luo Luo,et al.  Decentralized Stochastic Variance Reduced Extragradient Method , 2022, ArXiv.

[19]  Miao Yang,et al.  Client Selection for Federated Learning With Label Noise , 2022, IEEE Transactions on Vehicular Technology.

[20]  Byung Hyung Kim,et al.  ALIS: Learning Affective Causality Behind Daily Activities From a Wearable Life-Log System , 2021, IEEE Transactions on Cybernetics.

[21]  Balaji Lakshminarayanan,et al.  An instance-dependent simulation framework for learning with label noise , 2021, Machine Learning.

[22]  Ge Yang,et al.  Deep Neural Networks Learn Meta-Structures from Noisy Labels in Semantic Segmentation , 2021, AAAI.

[23]  Changick Kim,et al.  Robust Federated Learning With Noisy Labels , 2020, IEEE Intelligent Systems.

[24]  K. Ramchandran,et al.  An Efficient Framework for Clustered Federated Learning , 2020, IEEE Transactions on Information Theory.

[25]  Dejing Dou,et al.  Generalized Data Weighting via Class-level Gradient Manipulation , 2021, NeurIPS.

[26]  Hongbo Zhu,et al.  Client Selection Based on Label Quantity Information for Federated Learning , 2021, 2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC).

[27]  Suhas Diggavi,et al.  A Field Guide to Federated Optimization , 2021, ArXiv.

[28]  Bo Han,et al.  Federated Noisy Client Learning , 2021, IEEE transactions on neural networks and learning systems.

[29]  Hangyu Zhu,et al.  Federated Learning on Non-IID Data: A Survey , 2021, Neurocomputing.

[30]  Tengyu Ma,et al.  Label Noise SGD Provably Prefers Flat Global Minimizers , 2021, NeurIPS.

[31]  Andreas Keller,et al.  Swarm Learning for decentralized and confidential clinical machine learning , 2021, Nature.

[32]  Jonas Mueller,et al.  Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks , 2021, NeurIPS Datasets and Benchmarks.

[33]  Kin K. Leung,et al.  Overcoming Noisy and Irrelevant Data in Federated Learning , 2020, 2020 25th International Conference on Pattern Recognition (ICPR).

[34]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[35]  Isaac L. Chuang,et al.  Confident Learning: Estimating Uncertainty in Dataset Labels , 2019, J. Artif. Intell. Res..

[36]  Qinghua Liu,et al.  Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization , 2020, NeurIPS.

[37]  E Weinan,et al.  On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics , 2020, CSIAM Transactions on Applied Mathematics.

[38]  H. Vincent Poor,et al.  Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learning , 2020, 2020 IEEE 21st International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[39]  Xiangliang Zhang,et al.  Robust Federated Learning via Collaborative Machine Teaching , 2020, AAAI.

[40]  Victor C. M. Leung,et al.  Blockchain and Machine Learning for Communications and Networking Systems , 2020, IEEE Communications Surveys & Tutorials.

[41]  Hossein Mobahi,et al.  Fantastic Generalization Measures and Where to Find Them , 2019, ICLR.

[42]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning , 2019, ICML.

[43]  Tzu-Ming Harry Hsu,et al.  Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification , 2019, ArXiv.

[44]  Nenghai Yu,et al.  Capacity Control of ReLU Neural Networks by Basis-path Norm , 2018, AAAI.

[45]  Yongqiang Wang,et al.  ADMM Based Privacy-Preserving Decentralized Optimization , 2017, IEEE Transactions on Information Forensics and Security.

[46]  Dimosthenis Karatzas,et al.  On the Labeling Correctness in Computer Vision Datasets , 2018, IAL@PKDD/ECML.

[47]  Nathan Srebro,et al.  Exploring Generalization in Deep Learning , 2017, NIPS.

[48]  Gregory Cohen,et al.  EMNIST: Extending MNIST to handwritten letters , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[49]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[50]  Ruslan Salakhutdinov,et al.  Path-SGD: Path-Normalized Optimization in Deep Neural Networks , 2015, NIPS.

[51]  Xiaogang Wang,et al.  Learning from massive noisy labeled data for image classification , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[53]  A. Stephen Morse,et al.  Accelerated linear iterations for distributed averaging , 2011, Annu. Rev. Control..

[54]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[55]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .