Privacy for Free: Wireless Federated Learning via Uncoded Transmission With Adaptive Power Control

Federated Learning (FL) refers to distributed protocols that avoid direct raw data exchange among the participating devices while training for a common learning task. This way, FL can potentially reduce the information on the local data sets that is leaked via communications. In order to provide formal privacy guarantees, however, it is generally necessary to put in place additional masking mechanisms. When FL is implemented in wireless systems via uncoded transmission, the channel noise can directly act as a privacy-inducing mechanism. This paper demonstrates that, as long as the privacy constraint level, measured via differential privacy (DP), is below a threshold that decreases with the signal-to-noise ratio (SNR), uncoded transmission achieves privacy “for free”, i.e., without affecting the learning performance. More generally, this work studies adaptive power allocation (PA) for distributed gradient descent in wireless FL with the aim of minimizing the learning optimality gap under privacy and power constraints. Both orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA) transmission with “over-the-air-computing” are studied, and solutions are obtained in closed form for an offline optimization setting. Furthermore, heuristic online methods are proposed that leverage iterative one-step-ahead optimization. The importance of dynamic PA and the potential benefits of NOMA versus OMA are demonstrated through extensive simulations.

[1]  Raj Kumar Maity,et al.  vqSGD: Vector Quantized Stochastic Gradient Descent , 2019, IEEE Transactions on Information Theory.

[2]  Zhiwei Steven Wu,et al.  Understanding Gradient Clipping in Private SGD: A Geometric Perspective , 2020, NeurIPS.

[3]  Meixia Tao,et al.  Gradient Statistics Aware Power Control for Over-the-Air Federated Learning , 2020, IEEE Transactions on Wireless Communications.

[4]  H. Vincent Poor,et al.  Federated Learning With Differential Privacy: Algorithms and Performance Analysis , 2019, IEEE Transactions on Information Forensics and Security.

[5]  Mehdi Bennis,et al.  Wireless Network Intelligence at the Edge , 2018, Proceedings of the IEEE.

[6]  Sujith Ravi,et al.  Efficient On-Device Models using Neural Projections , 2019, ICML.

[7]  Stefano Rini,et al.  Efficient Federated Learning over Multiple Access Channel with Differential Privacy Constraints , 2020, ArXiv.

[8]  Mark W. Schmidt,et al.  Hybrid Deterministic-Stochastic Methods for Data Fitting , 2011, SIAM J. Sci. Comput..

[9]  Aryan Mokhtari,et al.  FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization , 2019, AISTATS.

[10]  X. Lin,et al.  Doppler Shift and Coherence Time of 5G Vehicular Channels at 3.5 GHz , 2018, 2018 IEEE International Symposium on Antennas and Propagation & USNC/URSI National Radio Science Meeting.

[11]  Michael Gastpar,et al.  Computation Over Multiple-Access Channels , 2007, IEEE Transactions on Information Theory.

[12]  Xu Chen,et al.  Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing , 2019, Proceedings of the IEEE.

[13]  Jo Verhaevert,et al.  RMS Delay Spread vs. Coherence Bandwidth from 5G Indoor Radio Channel Measurements at 3.5 GHz Band , 2020, Sensors.

[14]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[15]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[16]  Dan Alistarh,et al.  QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks , 2016, 1610.02132.

[17]  Mohammad Mohammadi Amiri,et al.  Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air , 2020 .

[18]  Kaibin Huang,et al.  Towards an Intelligent Edge: Wireless Communication Meets Machine Learning , 2018, ArXiv.

[19]  Kin K. Leung,et al.  Adaptive Federated Learning in Resource Constrained Edge Computing Systems , 2018, IEEE Journal on Selected Areas in Communications.

[20]  Mark W. Schmidt,et al.  Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition , 2016, ECML/PKDD.

[21]  Gilles Barthe,et al.  Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences , 2018, NeurIPS.

[22]  Kamyar Azizzadenesheli,et al.  signSGD: compressed optimisation for non-convex problems , 2018, ICML.

[23]  Kaibin Huang,et al.  High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning , 2019, 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[24]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[25]  Tong Zhang,et al.  Stochastic Optimization with Importance Sampling for Regularized Loss Minimization , 2014, ICML.

[26]  Kobi Cohen,et al.  On Analog Gradient Descent Learning Over Multiple Access Fading Channels , 2019, IEEE Transactions on Signal Processing.

[27]  Deniz Gündüz,et al.  Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air , 2019, 2019 IEEE International Symposium on Information Theory (ISIT).

[28]  Yue Zhao,et al.  Federated Learning with Non-IID Data , 2018, ArXiv.

[29]  Zhi Ding,et al.  Federated Learning via Over-the-Air Computation , 2018, IEEE Transactions on Wireless Communications.

[30]  Vitaly Shmatikov,et al.  Exploiting Unintended Feature Leakage in Collaborative Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[31]  Nan Wu,et al.  The Value of Collaboration in Convex Machine Learning with Differential Privacy , 2019, 2020 IEEE Symposium on Security and Privacy (SP).

[32]  Kaibin Huang,et al.  Optimal Power Control for Over-the-Air Computation , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[33]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[34]  Anit Kumar Sahu,et al.  Federated Optimization in Heterogeneous Networks , 2018, MLSys.

[35]  Gaurav Kapoor,et al.  Protection Against Reconstruction and Its Applications in Private Federated Learning , 2018, ArXiv.

[36]  Ravi Tandon,et al.  Communication Efficient Federated Learning over Multiple Access Channels , 2020, ArXiv.

[37]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[38]  Sanjiv Kumar,et al.  cpSGD: Communication-efficient and differentially-private distributed SGD , 2018, NeurIPS.

[39]  Deniz Gündüz,et al.  One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis , 2020, IEEE Transactions on Wireless Communications.

[40]  Aamir Mahmood,et al.  Time Synchronization in 5G Wireless Edge: Requirements and Solutions for Critical-MTC , 2019, IEEE Communications Magazine.

[41]  Kaibin Huang,et al.  High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning , 2020, IEEE Transactions on Signal Processing.

[42]  Kaibin Huang,et al.  Broadband Analog Aggregation for Low-Latency Federated Edge Learning , 2018, IEEE Transactions on Wireless Communications.

[43]  Masahiro Morikura,et al.  Differentially Private AirComp Federated Learning with Power Adaptation Harnessing Receiver Noise , 2020, GLOBECOM 2020 - 2020 IEEE Global Communications Conference.

[44]  Suvrit Sra,et al.  Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity , 2019, ICLR.

[45]  Richard Nock,et al.  Advances and Open Problems in Federated Learning , 2021, Found. Trends Mach. Learn..

[46]  Somesh Jha,et al.  Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures , 2015, CCS.

[47]  Ming Li,et al.  Wireless Federated Learning with Local Differential Privacy , 2020, 2020 IEEE International Symposium on Information Theory (ISIT).