Byzantine-Robust Federated Learning with Optimal Statistical Rates and Privacy Guarantees

We propose Byzantine-robust federated learning protocols with nearly optimal statistical rates. In contrast to prior work, our proposed protocols improve the dimension dependence and achieve a tight statistical rate in terms of all the parameters for strongly convex losses. We benchmark against competing protocols and show the empirical superiority of the proposed protocols. Finally, we remark that our protocols with bucketing can be naturally combined with privacy-guaranteeing procedures to introduce security against a semi-honest server. The code for evaluation is provided in https://github.com/wanglun1996/secure-robust-federated-learning .

[1]  Abhishek V A,et al.  Federated Learning: Collaborative Machine Learning without Centralized Training Data , 2022, international journal of engineering technology and management sciences.

[2]  Michael I. Jordan,et al.  Robust Estimation for Nonparametric Families via Generative Adversarial Networks , 2022, ArXiv.

[3]  Sai Praneeth Karimireddy,et al.  Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing , 2020, ICLR.

[4]  Qiang Wang,et al.  Data Poisoning Attacks on Federated Machine Learning , 2020, IEEE Internet of Things Journal.

[5]  Zaïd Harchaoui,et al.  Robust Aggregation for Federated Learning , 2019, IEEE Transactions on Signal Processing.

[6]  Banghua Zhu,et al.  Generalized Resilience and Robust Statistics , 2019, The Annals of Statistics.

[7]  Beng Chin Ooi,et al.  Feature Inference Attack on Model Predictions in Vertical Federated Learning , 2020, 2021 IEEE 37th International Conference on Data Engineering (ICDE).

[8]  A. Salman Avestimehr,et al.  Byzantine-Resilient Secure Federated Learning , 2020, IEEE Journal on Selected Areas in Communications.

[9]  Banghua Zhu,et al.  Robust estimation via generalized quasi-gradients , 2020, Information and Inference: A Journal of the IMA.

[10]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[11]  G. Lugosi,et al.  Robust multivariate mean estimation: The optimality of trimmed mean , 2019, The Annals of Statistics.

[12]  Learning to Attack Distributionally Robust Federated Learning , 2021 .

[13]  Tancrède Lepoint,et al.  Secure Single-Server Aggregation with (Poly)Logarithmic Overhead , 2020, IACR Cryptol. ePrint Arch..

[14]  Samuel B. Hopkins,et al.  Robust and Heavy-Tailed Mean Estimation Made Simple, via Regret Minimization , 2020, NeurIPS.

[15]  Ankit Pensia,et al.  Outlier Robust Mean Estimation with Subgaussian Rates via Stability , 2020, NeurIPS.

[16]  Kartik Sreenivasan,et al.  Attack of the Tails: Yes, You Really Can Backdoor Federated Learning , 2020, NeurIPS.

[17]  L. Golubchik,et al.  Backdoor Attacks on Federated Meta-Learning , 2020, ArXiv.

[18]  Sai Praneeth Karimireddy,et al.  Secure Byzantine-Robust Machine Learning , 2020, ArXiv.

[19]  Bo Li,et al.  DBA: Distributed Backdoor Attacks against Federated Learning , 2020, ICLR.

[20]  Yuan Yao,et al.  Generative Adversarial Nets for Robust Scatter Estimation: A Proper Scoring Rule Perspective , 2019, J. Mach. Learn. Res..

[21]  David Tse,et al.  Deconstructing Generative Adversarial Networks , 2019, IEEE Transactions on Information Theory.

[22]  Vitaly Shmatikov,et al.  How To Backdoor Federated Learning , 2018, AISTATS.

[23]  Ivan Beschastnikh,et al.  The Limitations of Federated Learning in Sybil Settings , 2020, RAID.

[24]  Ananda Theertha Suresh,et al.  Can You Really Backdoor Federated Learning? , 2019, ArXiv.

[25]  Bo Li,et al.  Attack-Resistant Federated Learning with Residual-based Reweighting , 2019, ArXiv.

[26]  Daniel M. Kane,et al.  Nearly Tight Bounds for Robust Proper Learning of Halfspaces with a Margin , 2019, NeurIPS.

[27]  Samuel B. Hopkins,et al.  Quantum Entropy Scoring for Fast Robust Mean Estimation and Improved Outlier Detection , 2019, NeurIPS.

[28]  Amir Houmansadr,et al.  Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[29]  Yang Song,et al.  Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning , 2018, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[30]  Prateek Mittal,et al.  Analyzing Federated Learning through an Adversarial Lens , 2018, ICML.

[31]  Yu Cheng,et al.  High-Dimensional Robust Mean Estimation in Nearly-Linear Time , 2018, SODA.

[32]  Kannan Ramchandran,et al.  Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning , 2018, ICML.

[33]  Ilias Diakonikolas,et al.  Efficient Algorithms and Lower Bounds for Robust Linear Regression , 2018, SODA.

[34]  Jerry Li,et al.  Sever: A Robust Meta-Algorithm for Stochastic Optimization , 2018, ICML.

[35]  G. Lugosi,et al.  Sub-Gaussian estimators of the mean of a random vector , 2017, The Annals of Statistics.

[36]  J. Steinhardt.  Lecture Notes for STAT260 (Robust Statistics) , 2019 .

[37]  Yuan Yao,et al.  Robust Estimation and Generative Adversarial Nets , 2018, ArXiv.

[38]  Ivan Beschastnikh,et al.  Mitigating Sybils in Federated Learning Poisoning , 2018, ArXiv.

[39]  Dimitris S. Papailiopoulos,et al.  DRACO: Byzantine-resilient Distributed Training via Redundant Gradients , 2018, ICML.

[40]  Dan Alistarh,et al.  Byzantine Stochastic Gradient Descent , 2018, NeurIPS.

[41]  Kannan Ramchandran,et al.  Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates , 2018, ICML.

[42]  Rachid Guerraoui,et al.  The Hidden Vulnerability of Distributed Learning in Byzantium , 2018, ICML.

[43]  Gregory Valiant,et al.  Resilience: A Criterion for Learning in the Presence of Arbitrary Outliers , 2017, ITCS.

[44]  Jerry Zheng Li,et al.  Principled approaches to robust machine learning and beyond , 2018 .

[45]  Jacob Steinhardt,et al.  ROBUST LEARNING: INFORMATION THEORY AND ALGORITHMS A DISSERTATION SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY , 2018 .

[46]  Rachid Guerraoui,et al.  Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent , 2017, NIPS.

[47]  Matthieu Lerasle,et al.  ROBUST MACHINE LEARNING BY MEDIAN-OF-MEANS: THEORY AND PRACTICE , 2019 .

[48]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[49]  Percy Liang,et al.  Certified Defenses for Data Poisoning Attacks , 2017, NIPS.

[50]  Jerry Li,et al.  Being Robust (in High Dimensions) Can Be Practical , 2017, ICML.

[51]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[52]  Mengjie Chen,et al.  A general decision theory for Huber’s (cid:2) -contamination model , 2017 .

[53]  Daniel M. Kane,et al.  Robust Estimators in High Dimensions without the Computational Intractability , 2016, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS).

[54]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[55]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[56]  P. J. Huber.  Robust Regression: Asymptotics, Conjectures and Monte Carlo , 1973 .