Vulnerabilities in Federated Learning

With more regulations tackling the protection of users’ privacy-sensitive data in recent years, access to such data has become increasingly restricted. A new decentralized training paradigm, known as Federated Learning (FL), enables multiple clients located at different geographical locations to learn a machine learning model collaboratively without sharing their data. While FL has recently emerged as a promising solution to preserve users’ privacy, this new paradigm’s potential security implications may hinder its widespread adoption. The existing FL protocols exhibit new unique vulnerabilities that adversaries can exploit to compromise the trained model. FL is often preferred in learning environments where security and privacy are the key concerns. Therefore, it is crucial to raise awareness of the consequences resulting from the new threats to FL systems. To date, the security of traditional machine learning systems has been widely examined. However, many open challenges and complex questions are still surrounding FL security. In this paper, we bridge the gap in FL literature by providing a comprehensive survey of the unique security vulnerabilities exposed by the FL ecosystem. We highlight the vulnerabilities sources, key attacks on FL, defenses, as well as their unique challenges, and discuss promising future research directions towards more robust FL.

[1]  Silvio Micali,et al.  The knowledge complexity of interactive proof-systems , 1985, STOC '85.

[2]  Blaine Nelson,et al.  Exploiting Machine Learning to Subvert Your Spam Filter , 2008, LEET.

[3]  Blaine Nelson,et al.  The security of machine learning , 2010, Machine Learning.

[4]  Thomas Plantard,et al.  Reaction Attack on Outsourced Computing with Fully Homomorphic Encryption Schemes , 2011, ICISC.

[5]  Fabio Roli,et al.  Evasion Attacks against Machine Learning at Test Time , 2013, ECML/PKDD.

[6]  Craig Gentry,et al.  Pinocchio: Nearly Practical Verifiable Computation , 2013, 2013 IEEE Symposium on Security and Privacy.

[7]  A. Anonymous,et al.  Consumer Data Privacy in a Networked World: A Framework for Protecting Privacy and Promoting Innovation in the Global Digital Economy , 2013, J. Priv. Confidentiality.

[8]  Richard Colbaugh,et al.  Moving target defense for adaptive adversaries , 2013, 2013 IEEE International Conference on Intelligence and Security Informatics.

[9]  Minghui Zhu,et al.  Comparing Different Moving Target Defense Techniques , 2014, MTD '14.

[10]  Julian Jang,et al.  A survey of emerging threats in cybersecurity , 2014, J. Comput. Syst. Sci..

[11]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[12]  Prateek Saxena,et al.  Auror: defending against poisoning attacks in collaborative deep learning systems , 2016, ACSAC.

[13]  Peter Richtárik,et al.  Federated Optimization: Distributed Machine Learning for On-Device Intelligence , 2016, ArXiv.

[14]  Sebastian Nowozin,et al.  Oblivious Multi-Party Machine Learning on Trusted Processors , 2016, USENIX Security Symposium.

[15]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[16]  S. Mercy Shalinie,et al.  A survey of distributed denial of service attack , 2016, 2016 10th International Conference on Intelligent Systems and Control (ISCO).

[17]  Omar Alfandi,et al.  Analysis of cloud computing attacks and countermeasures , 2016, 2016 18th International Conference on Advanced Communication Technology (ICACT).

[18]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[19]  Fabio Roli,et al.  Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization , 2017, AISec@CCS.

[20]  Payman Mohassel,et al.  SecureML: A System for Scalable Privacy-Preserving Machine Learning , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[21]  Giuseppe Ateniese,et al.  Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning , 2017, CCS.

[22]  Brendan Dolan-Gavitt,et al.  BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain , 2017, ArXiv.

[23]  Rachid Guerraoui,et al.  Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent , 2017, NIPS.

[24]  Paul Voigt,et al.  The Eu General Data Protection Regulation (Gdpr): A Practical Guide , 2017 .

[25]  Tassilo Klein,et al.  Differentially Private Federated Learning: A Client Level Perspective , 2017, ArXiv.

[26]  G. Sahoo,et al.  Man-in-the-middle attack in wireless and computer networking — A review , 2017, 2017 3rd International Conference on Advances in Computing,Communication & Automation (ICACCA) (Fall).

[27]  Srinivas Devadas,et al.  A Formal Foundation for Secure Remote Execution of Enclaves , 2017, IACR Cryptol. ePrint Arch..

[28]  Dawn Xiaodong Song,et al.  Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning , 2017, ArXiv.

[29]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[30]  Sarvar Patel,et al.  Practical Secure Aggregation for Privacy-Preserving Machine Learning , 2017, IACR Cryptol. ePrint Arch..

[31]  Martín Abadi,et al.  Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data , 2016, ICLR.

[32]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[33]  Cheng Lei,et al.  Moving Target Defense Techniques: A Survey , 2018, Secur. Commun. Networks.

[34]  Wen-Chuan Lee,et al.  Trojaning Attack on Neural Networks , 2018, NDSS.

[35]  Kannan Ramchandran,et al.  Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates , 2018, ICML.

[36]  Gaurav Kapoor,et al.  Protection Against Reconstruction and Its Applications in Private Federated Learning , 2018, ArXiv.

[37]  Brendan Dolan-Gavitt,et al.  Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks , 2018, RAID.

[38]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[39]  Quoc V. Le,et al.  Don't Decay the Learning Rate, Increase the Batch Size , 2017, ICLR.

[40]  Dinesh C. Verma,et al.  Distributed AI and security issues in federated environments , 2018, ICDCN Workshops.

[41]  Sebastian Caldas,et al.  Expanding the Reach of Federated Learning by Reducing Client Resource Requirements , 2018, ArXiv.

[42]  Tudor Dumitras,et al.  Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks , 2018, NeurIPS.

[43]  Chang Liu,et al.  Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[44]  Wouter Joosen,et al.  Chained Anomaly Detection Models for Federated Learning: An Intrusion Detection Case Study , 2018, Applied Sciences.

[45]  Aleksander Madry,et al.  Exploring the Landscape of Spatial Robustness , 2017, ICML.

[46]  Junpu Wang,et al.  FedMD: Heterogenous Federated Learning via Model Distillation , 2019, ArXiv.

[47]  Shiho Moriai,et al.  Privacy-Preserving Deep Learning via Additively Homomorphic Encryption , 2019, 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH).

[48]  Hamed Haddadi,et al.  Efficient and Private Federated Learning using TEE , 2019 .

[49]  Jörn-Henrik Jacobsen,et al.  Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness , 2019, ArXiv.

[50]  Yi Sun,et al.  Testing Robustness Against Unforeseen Adversaries , 2019, ArXiv.

[51]  Tara Javidi,et al.  Peer-to-peer Federated Learning on Graphs , 2019, ArXiv.

[52]  Vitaly Shmatikov,et al.  Exploiting Unintended Feature Leakage in Collaborative Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[53]  Suman Jana,et al.  Certified Robustness to Adversarial Examples with Differential Privacy , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[54]  Alan L. Yuille,et al.  Feature Denoising for Improving Adversarial Robustness , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Bing Chen,et al.  Poisoning Attack in Federated Learning using Generative Adversarial Nets , 2019, 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE).

[56]  Song Han,et al.  Deep Leakage from Gradients , 2019, NeurIPS.

[57]  Abnormal Client Behavior Detection in Federated Learning , 2019, ArXiv.

[58]  Justin Hsu,et al.  Data Poisoning against Differentially-Private Learners: Attacks and Defenses , 2019, IJCAI.

[59]  Rui Zhang,et al.  A Hybrid Approach to Privacy-Preserving Federated Learning , 2018, Informatik Spektrum.

[60]  Prateek Mittal,et al.  Analyzing Federated Learning through an Adversarial Lens , 2018, ICML.

[61]  Leandros Tassiulas,et al.  Model Pruning Enables Efficient Federated Learning on Edge Devices , 2019, IEEE transactions on neural networks and learning systems.

[62]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[63]  Sebastian U. Stich,et al.  Local SGD Converges Fast and Communicates Little , 2018, ICLR.

[64]  Bo Li,et al.  Attack-Resistant Federated Learning with Residual-based Reweighting , 2019, ArXiv.

[65]  Nassir Navab,et al.  BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning , 2019, ArXiv.

[66]  Johan Karlsson,et al.  Applications of Deep-Learning in Exploiting Large-Scale and Heterogeneous Compound Data in Industrial Pharmaceutical Research , 2019, Front. Pharmacol..

[67]  Tianjian Chen,et al.  Federated Machine Learning: Concept and Applications , 2019 .

[68]  Moran Baruch,et al.  A Little Is Enough: Circumventing Defenses For Distributed Learning , 2019, NeurIPS.

[69]  Min Du,et al.  Free-riders in Federated Learning: Attacks and Defenses , 2019, ArXiv.

[70]  Dan Boneh,et al.  Adversarial Training and Robustness for Multiple Perturbations , 2019, NeurIPS.

[71]  James Bailey,et al.  On the Convergence and Robustness of Adversarial Training , 2021, ICML.

[72]  Takayuki Nishio,et al.  Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge , 2018, ICC 2019 - 2019 IEEE International Conference on Communications (ICC).

[73]  Ananda Theertha Suresh,et al.  Can You Really Backdoor Federated Learning? , 2019, ArXiv.

[74]  Amir Houmansadr,et al.  Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[75]  Surya Nepal,et al.  End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things , 2020, 2020 International Symposium on Reliable Distributed Systems (SRDS).

[76]  Jinyuan Jia,et al.  Local Model Poisoning Attacks to Byzantine-Robust Federated Learning , 2019, USENIX Security Symposium.

[77]  Kartik Sreenivasan,et al.  Attack of the Tails: Yes, You Really Can Backdoor Federated Learning , 2020, NeurIPS.

[78]  Bo Zhao,et al.  iDLG: Improved Deep Leakage from Gradients , 2020, ArXiv.

[79]  Daniel Rueckert,et al.  Robust Aggregation for Adaptive Privacy Preserving Federated Learning in Healthcare , 2020, ArXiv.

[80]  Vitaly Shmatikov,et al.  How To Backdoor Federated Learning , 2018, AISTATS.

[81]  Zhou Chuanxin,et al.  Federated Learning with Gaussian Differential Privacy , 2020, RICAI.

[82]  Han Yu,et al.  FedCoin: A Peer-to-Peer Payment System for Federated Learning , 2020, Federated Learning.

[83]  Anbu Huang,et al.  Dynamic backdoor attacks against federated learning , 2020, ArXiv.

[84]  Tianjian Chen,et al.  Learning to Detect Malicious Clients for Robust Federated Learning , 2020, ArXiv.

[85]  Lingfei Wu,et al.  Towards Byzantine-Resilient Federated Learning via Group-Wise Robust Aggregation , 2020, Federated Learning.

[86]  Bo Li,et al.  DBA: Distributed Backdoor Attacks against Federated Learning , 2020, ICLR.

[87]  Mario Polino,et al.  Evasion Attacks against Banking Fraud Detection Systems , 2020, RAID.

[88]  Douglas A. Talbert,et al.  Model Evasion Attack on Intrusion Detection Systems using Adversarial Machine Learning , 2020, 2020 54th Annual Conference on Information Sciences and Systems (CISS).

[89]  Wei Yang Bryan Lim,et al.  Federated Learning in Mobile Edge Networks: A Comprehensive Survey , 2019, IEEE Communications Surveys & Tutorials.

[90]  Zhifei Zhang,et al.  Analyzing User-Level Privacy Attack Against Federated Learning , 2020, IEEE Journal on Selected Areas in Communications.

[91]  Shouling Ji,et al.  Privacy Leakage of Real-World Vertical Federated Learning , 2020, ArXiv.

[92]  On Safeguarding Privacy and Security in the Framework of Federated Learning , 2019, IEEE Network.

[93]  Jean-Pierre Hubaux,et al.  Drynx: Decentralized, Secure, Verifiable System for Statistical Queries and Machine Learning on Distributed Datasets , 2019, IEEE Transactions on Information Forensics and Security.

[94]  Ximeng Liu,et al.  Privacy-Preserving in Defending against Membership Inference Attacks , 2020 .

[95]  Sailik Sengupta,et al.  A Survey of Moving Target Defenses for Network Security , 2019, IEEE Communications Surveys & Tutorials.

[96]  Tianrui Li,et al.  Fairness and Accuracy in Federated Learning , 2020, ArXiv.

[97]  Yanzhao Wu,et al.  A Framework for Evaluating Gradient Leakage Attacks in Federated Learning , 2020, ArXiv.

[98]  Yasaman Khazaeni,et al.  Federated Learning with Matched Averaging , 2020, ICLR.

[99]  Li Chen,et al.  Robust Federated Learning With Noisy Communication , 2019, IEEE Transactions on Communications.

[100]  Seong-Lyun Kim,et al.  Federated Knowledge Distillation , 2020, ArXiv.

[101]  Tao Xiang,et al.  A training-integrity privacy-preserving federated learning scheme with trusted execution environment , 2020, Inf. Sci..

[102]  Ahmed M. Abdelmoniem,et al.  Compressed Communication for Distributed Deep Learning: Survey and Quantitative Evaluation , 2020 .

[103]  Sashank J. Reddi,et al.  SCAFFOLD: Stochastic Controlled Averaging for Federated Learning , 2019, ICML.

[104]  Micah J. Sheller,et al.  The future of digital health with federated learning , 2020, npj Digital Medicine.

[105]  Rui Li,et al.  Online Federated Multitask Learning , 2020 .

[106]  Rongxing Lu,et al.  Adaptive privacy-preserving federated learning , 2020, Peer-to-Peer Networking and Applications.

[107]  Yang Liu,et al.  BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning , 2020, USENIX ATC.

[108]  Anit Kumar Sahu,et al.  Federated Optimization in Heterogeneous Networks , 2018, MLSys.

[109]  A. Borovik,et al.  Homomorphic Encryption and Some Black Box Attacks , 2020, ICMS.

[110]  Yan Zhang,et al.  Blockchain and Federated Learning for Privacy-Preserved Data Sharing in Industrial IoT , 2020, IEEE Transactions on Industrial Informatics.

[111]  Yanyang Lu,et al.  An Efficient and Robust Aggregation Algorithm for Learning Federated CNN , 2020, SPML.

[112]  Han Yu,et al.  Threats to Federated Learning: A Survey , 2020, ArXiv.

[113]  Ahmed M. Abdelmoniem,et al.  On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning , 2019, AAAI.

[114]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2019, Found. Trends Mach. Learn..

[115]  Virginia Smith,et al.  Ditto: Fair and Robust Federated Learning Through Personalization , 2020, ICML.

[116]  Ali Dehghantanha,et al.  A survey on security and privacy of federated learning , 2021, Future Gener. Comput. Syst..

[117]  Mohsen Guizani,et al.  A Survey on Federated Learning: The Journey From Centralized to Distributed On-Site Learning and Beyond , 2021, IEEE Internet of Things Journal.

[118]  Shafkat Islam,et al.  DeSMP: Differential Privacy-exploited Stealthy Model Poisoning Attacks in Federated Learning , 2021, 2021 17th International Conference on Mobility, Sensing and Networking (MSN).

[119]  Peter B. Walker,et al.  Federated Learning for Healthcare Informatics , 2019, Journal of Healthcare Informatics Research.

[120]  Richard Vidal,et al.  Free-rider Attacks on Model Aggregation in Federated Learning , 2020, AISTATS.

[121]  Ivan Beschastnikh,et al.  Biscotti: A Blockchain System for Private and Secure Federated Learning , 2021, IEEE Transactions on Parallel and Distributed Systems.

[122]  Farinaz Koushanfar,et al.  A Taxonomy of Attacks on Federated Learning , 2021, IEEE Security & Privacy.

[123]  Ying-Chang Liang,et al.  Toward Smart Security Enhancement of Federated Learning Networks , 2020, IEEE Network.

[124]  Surya Nepal,et al.  Man-in-the-Middle Attacks Against Machine Learning Classifiers Via Malicious Generative Models , 2019, IEEE Transactions on Dependable and Secure Computing.

[125]  Zaïd Harchaoui,et al.  Robust Aggregation for Federated Learning , 2019, IEEE Transactions on Signal Processing.