Towards Communication-Efficient and Attack-Resistant Federated Edge Learning for Industrial Internet of Things

Federated Edge Learning (FEL) allows edge nodes to train a global deep learning model collaboratively for edge computing in the Industrial Internet of Things (IIoT), which significantly promotes the development of Industrial 4.0. However, FEL faces two critical challenges: communication overhead and data privacy. FEL suffers from expensive communication overhead when training large-scale multi-node models. Furthermore, due to the vulnerability of FEL to gradient leakage and label-flipping attacks, the training process of the global model is easily compromised by adversaries. To address these challenges, we propose a communication-efficient and privacy-enhanced asynchronous FEL framework for edge computing in IIoT. First, we introduce an asynchronous model update scheme to reduce the computation time that edge nodes wait for global model aggregation. Second, we propose an asynchronous local differential privacy mechanism, which improves communication efficiency and mitigates gradient leakage attacks by adding well-designed noise to the gradients of edge nodes. Third, we design a cloud-side malicious node detection mechanism to detect malicious nodes by testing the local model quality. Such a mechanism can avoid malicious nodes participating in training to mitigate label-flipping attacks. Extensive experimental studies on two real-world datasets demonstrate that the proposed framework can not only improve communication efficiency but also mitigate malicious attacks while its accuracy is comparable to traditional FEL frameworks.

[1]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[2]  Ling Liu,et al.  Preserving data privacy in outsourcing data aggregation services , 2007, TOIT.

[3]  Kunal Talwar,et al.  Mechanism Design via Differential Privacy , 2007, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[4]  Cynthia Dwork,et al.  Differential Privacy: A Survey of Results , 2008, TAMC.

[5]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[6]  Peter J. Haas,et al.  Large-scale matrix factorization with distributed stochastic gradient descent , 2011, KDD.

[7]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[8]  Pramod Viswanath,et al.  Extremal Mechanisms for Local Differential Privacy , 2014, J. Mach. Learn. Res..

[9]  Cong Wang,et al.  Enabling Privacy-Preserving Image-Centric Social Discovery , 2014, 2014 IEEE 34th International Conference on Distributed Computing Systems.

[10]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[11]  Elisa Bertino,et al.  Internet of Things (IoT) , 2016, ACM Trans. Internet Techn..

[12]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[13]  Dan Alistarh,et al.  QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks , 2016, 1610.02132.

[14]  Rachid Guerraoui,et al.  Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent , 2017, NIPS.

[15]  Tassilo Klein,et al.  Differentially Private Federated Learning: A Client Level Perspective , 2017, ArXiv.

[16]  Ji Liu,et al.  Gradient Sparsification for Communication-Efficient Distributed Optimization , 2017, NeurIPS.

[17]  Dan Alistarh,et al.  Byzantine Stochastic Gradient Descent , 2018, NeurIPS.

[18]  Dan Alistarh,et al.  The Convergence of Sparsified Gradient Methods , 2018, NeurIPS.

[19]  Martin Jaggi,et al.  Sparsified SGD with Memory , 2018, NeurIPS.

[20]  Marco Scavuzzo,et al.  Asynchronous Federated Learning for Geospatial Applications , 2018, DMLE/IOTSTREAMING@PKDD/ECML.

[21]  Daniel Rueckert,et al.  A generic framework for privacy preserving deep learning , 2018, ArXiv.

[22]  William J. Dally,et al.  Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training , 2017, ICLR.

[23]  Reza Shokri,et al.  Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks , 2018, ArXiv.

[24]  Dimitris S. Papailiopoulos,et al.  ATOMO: Communication-efficient Learning via Atomic Sparsification , 2018, NeurIPS.

[25]  Samuel Marchal,et al.  DÏoT: A Federated Self-learning Anomaly Detection System for IoT , 2018, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

[26]  Sebastian U. Stich,et al.  Stochastic Distributed Learning with Gradient Quantization and Variance Reduction , 2019, 1904.05115.

[27]  Song Han,et al.  Deep Leakage from Gradients , 2019, NeurIPS.

[28]  Abnormal Client Behavior Detection in Federated Learning , 2019, ArXiv.

[29]  Prateek Mittal,et al.  Analyzing Federated Learning through an Adversarial Lens , 2018, ICML.

[30]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[31]  Indranil Gupta,et al.  Asynchronous Federated Optimization , 2019, ArXiv.

[32]  Shengli Xie,et al.  Incentive Mechanism for Reliable Federated Learning: A Joint Optimization Approach to Combining Reputation and Contract Theory , 2019, IEEE Internet of Things Journal.

[33]  Shaohuai Shi,et al.  A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification , 2019, IJCAI.

[34]  Jie Xu,et al.  The Tradeoff Between Privacy and Accuracy in Anomaly Detection Using Federated XGBoost , 2019, ArXiv.

[35]  Wei Li,et al.  A Dynamic Service Migration Mechanism in Edge Cognitive Computing , 2018, ACM Trans. Internet Techn..

[36]  James J. Q. Yu,et al.  PPGAN: Privacy-Preserving Generative Adversarial Network , 2019, 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS).

[37]  M. Shamim Hossain,et al.  Emotion recognition using secure edge and cloud computing , 2019, Inf. Sci..

[38]  Qing Ling,et al.  RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets , 2018, AAAI.

[39]  Hongyi Wang,et al.  DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation , 2019, NeurIPS.

[40]  M. Shamim Hossain,et al.  Smart-Edge-CoCaCo: AI-Enabled Smart Edge with Joint Computation, Caching, and Communication in Heterogeneous IoT , 2019, IEEE Network.

[41]  M. Shamim Hossain,et al.  An Audio-Visual Emotion Recognition System Using Deep Learning Fusion for a Cognitive Wireless Framework , 2019, IEEE Wireless Communications.

[42]  Rachid Guerraoui,et al.  AGGREGATHOR: Byzantine Machine Learning via Robust Gradient Aggregation , 2019, SysML.

[43]  Jinyuan Jia,et al.  Local Model Poisoning Attacks to Byzantine-Robust Federated Learning , 2019, USENIX Security Symposium.

[44]  Qing Ling,et al.  Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks , 2019, IEEE Transactions on Signal Processing.

[45]  Vitaly Shmatikov,et al.  How To Backdoor Federated Learning , 2018, AISTATS.

[46]  Yaochu Jin,et al.  Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[47]  Pan Hui,et al.  Privacy-Preserving Asynchronous Federated Learning Mechanism for Edge Network Computing , 2020, IEEE Access.

[48]  Suvrit Sra,et al.  Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity , 2019, ICLR.

[49]  Ke Zhang,et al.  Blockchain Empowered Asynchronous Federated Learning for Secure Data Sharing in Internet of Vehicles , 2020, IEEE Transactions on Vehicular Technology.

[50]  Yan Zhang,et al.  Differentially Private Asynchronous Federated Learning for Mobile Edge Computing in Urban Informatics , 2020, IEEE Transactions on Industrial Informatics.

[51]  Mei Wang,et al.  An Efficient Service Function Chaining Placement Algorithm in Mobile Edge Computing , 2020, ACM Trans. Internet Techn..

[52]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[53]  Ivan Beschastnikh,et al.  The Limitations of Federated Learning in Sybil Settings , 2020, RAID.

[54]  Mehmet Emre Gursoy,et al.  Data Poisoning Attacks Against Federated Learning Systems , 2020, ESORICS.

[55]  Mohsen Guizani,et al.  Reliable Federated Learning for Mobile Networks , 2019, IEEE Wireless Communications.

[56]  Abdullah M. Iliyasu,et al.  A Secure Federated Learning Framework for 5G Networks , 2020, IEEE Wireless Communications.

[57]  G. Faraci,et al.  Fog in the Clouds , 2020, ACM Trans. Internet Techn..

[58]  Suhas Diggavi,et al.  Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations , 2019, IEEE Journal on Selected Areas in Information Theory.

[59]  Yanzhao Wu,et al.  A Framework for Evaluating Gradient Leakage Attacks in Federated Learning , 2020, ArXiv.

[60]  Syed Hassan Ahmed,et al.  Dominant Data Set Selection Algorithms for Electricity Consumption Time-Series Data Analysis Based on Affine Transformation , 2019, IEEE Internet of Things Journal.

[61]  Privacy-Preserving Traffic Flow Prediction: A Federated Learning Approach , 2020, IEEE Internet of Things Journal.

[62]  Dusit Niyato,et al.  Federated learning for 6G communications: Challenges, methods, and future directions , 2020, China Communications.

[63]  Yan Zhang,et al.  Blockchain and Federated Learning for Privacy-Preserved Data Sharing in Industrial IoT , 2020, IEEE Transactions on Industrial Informatics.

[64]  M. Shamim Hossain,et al.  Deep Anomaly Detection for Time-Series Data in Industrial IoT: A Communication-Efficient On-Device Federated Learning Approach , 2020, IEEE Internet of Things Journal.

[65]  Shielding Collaborative Learning: Mitigating Poisoning Attacks Through Client-Side Detection , 2019, IEEE Transactions on Dependable and Secure Computing.