Communication and Energy Efficient Wireless Federated Learning with Intrinsic Privacy

Federated Learning (FL) is a collaborative learning framework that enables edge devices to collaboratively learn a global model while keeping raw data locally. Although FL avoids leaking direct information from local datasets, sensitive information can still be inferred from the shared models. To address the privacy issue in FL, differential privacy (DP) mechanisms are leveraged to provide formal privacy guarantee. However, when deploying FL at the wireless edge with over-the-air computation, ensuring client-level DP faces significant challenges. In this paper, we propose a novel wireless FL scheme called private federated edge learning with sparsification (PFELS) to provide client-level DP guarantee with intrinsic channel noise while reducing communication and energy overhead and improving model accuracy. The key idea of PFELS is for each device to first compress its model update and then adaptively design the transmit power of the compressed model update according to the wireless channel status without any artificial noise addition. We provide a privacy analysis for PFELS and prove the convergence of PFELS under general non-convex and non-IID settings. Experimental results show that compared with prior work, PFELS can improve the accuracy with the same DP guarantee and save communication and energy costs simultaneously.

[1]  Xi Sheryl Zhang,et al.  Differentially Private Federated Learning with Local Regularization and Sparsification , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Yuanxiong Guo,et al.  Federated Learning with Sparsified Model Perturbation: Improving Accuracy under Client-Level Differential Privacy , 2022, IEEE Transactions on Mobile Computing.

[3]  Stefano Rini,et al.  Wireless Federated Learning with Limited Communication and Differential Privacy , 2021, 2021 IEEE Global Communications Conference (GLOBECOM).

[4]  Ravi Tandon,et al.  Privacy Amplification for Federated Learning via User Sampling and Wireless Aggregation , 2021, 2021 IEEE International Symposium on Information Theory (ISIT).

[5]  Deniz Gündüz,et al.  Federated Edge Learning with Misaligned Over-The-Air Computation , 2021, 2021 IEEE 22nd International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).

[6]  P. Kairouz,et al.  The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation , 2021, ICML.

[7]  H. Vincent Poor,et al.  User-Level Privacy-Preserving Federated Learning: Analysis and Performance Optimization , 2021, IEEE Transactions on Mobile Computing.

[8]  Miao Pan,et al.  Toward Energy-Efficient Federated Learning Over 5G+ Mobile Devices , 2021, IEEE Wireless Communications.

[9]  Yuqing Zhu,et al.  Voting-based Approaches For Differentially Private Federated Learning , 2020, ArXiv.

[10]  Yuanxiong Guo,et al.  Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization , 2020, IJCAI.

[11]  Úlfar Erlingsson,et al.  Tempered Sigmoid Activations for Deep Learning with Differential Privacy , 2020, AAAI.

[12]  Qinghua Liu,et al.  Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization , 2020, NeurIPS.

[13]  Kin K. Leung,et al.  Energy-Efficient Resource Management for Federated Edge Learning With CPU-GPU Heterogeneous Computing , 2020, IEEE Transactions on Wireless Communications.

[14]  Chaouki Ben Issaid,et al.  Harnessing Wireless Channels for Scalable and Privacy-Preserving Federated Learning , 2020, IEEE Transactions on Communications.

[15]  Osvaldo Simeone,et al.  Privacy for Free: Wireless Federated Learning via Uncoded Transmission With Adaptive Power Control , 2020, IEEE Journal on Selected Areas in Communications.

[16]  Rui Hu,et al.  Personalized Federated Learning With Differential Privacy , 2020, IEEE Internet of Things Journal.

[17]  Masahiro Morikura,et al.  Differentially Private AirComp Federated Learning with Power Adaptation Harnessing Receiver Noise , 2020, GLOBECOM 2020 - 2020 IEEE Global Communications Conference.

[18]  Masatoshi Yoshikawa,et al.  FedSel: Federated SGD under Local Differential Privacy with Top-k Dimension Selection , 2020, DASFAA.

[19]  Martin Jaggi,et al.  A Unified Theory of Decentralized SGD with Changing Topology and Local Updates , 2020, ICML.

[20]  Meixia Tao,et al.  Gradient Statistics Aware Power Control for Over-the-Air Federated Learning , 2020, IEEE Transactions on Wireless Communications.

[21]  Manzil Zaheer,et al.  Adaptive Federated Optimization , 2020, ICLR.

[22]  Ming Li,et al.  Wireless Federated Learning with Local Differential Privacy , 2020, 2020 IEEE International Symposium on Information Theory (ISIT).

[23]  H. Vincent Poor,et al.  Federated Learning With Differential Privacy: Algorithms and Performance Analysis , 2019, IEEE Transactions on Information Forensics and Security.

[24]  Kobi Cohen,et al.  On Analog Gradient Descent Learning Over Multiple Access Fading Channels , 2019, IEEE Transactions on Signal Processing.

[25]  Deniz Gündüz,et al.  Federated Learning Over Wireless Fading Channels , 2019, IEEE Transactions on Wireless Communications.

[26]  Deniz Gündüz,et al.  Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[27]  Ji Liu,et al.  DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression , 2019, ICML.

[28]  Martin Jaggi,et al.  Error Feedback Fixes SignSGD and other Gradient Compression Schemes , 2019, ICML.

[29]  Deniz Gündüz,et al.  Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air , 2019, IEEE Transactions on Signal Processing.

[30]  Zhi Ding,et al.  Federated Learning via Over-the-Air Computation , 2018, IEEE Transactions on Wireless Communications.

[31]  Kaibin Huang,et al.  Broadband Analog Aggregation for Low-Latency Federated Edge Learning , 2018, IEEE Transactions on Wireless Communications.

[32]  Borja Balle,et al.  Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences , 2018, NeurIPS.

[33]  Mario Fritz,et al.  ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models , 2018, NDSS.

[34]  Tassilo Klein,et al.  Differentially Private Federated Learning: A Client Level Perspective , 2017, ArXiv.

[35]  William J. Dally,et al.  Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training , 2017, ICLR.

[36]  H. Brendan McMahan,et al.  Learning Differentially Private Recurrent Language Models , 2017, ICLR.

[37]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[38]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[39]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Somesh Jha,et al.  Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures , 2015, CCS.

[42]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[43]  Ying Sun,et al.  Hybrid Local SGD for Federated Learning with Heterogeneous Communications , 2022, ICLR.

[44]  Yuanxiong Guo,et al.  Concentrated Differentially Private Federated Learning With Performance Analysis , 2021, IEEE Open Journal of the Computer Society.