FLORAS: Differentially Private Wireless Federated Learning Using Orthogonal Sequences

We propose a novel privacy-preserving uplink over-the-air computation (AirComp) method, termed FLORAS, for single-input single-output (SISO) wireless federated learning (FL) systems. From the communication design perspective, FLORAS eliminates the requirement of channel state information at the transmitters (CSIT) by leveraging the properties of orthogonal sequences. From the privacy perspective, we prove that FLORAS can offer both item-level and client-level differential privacy (DP) guarantees. Moreover, by adjusting the system parameters, FLORAS can flexibly achieve different DP levels at no additional cost. A novel FL convergence bound is derived which, combined with the privacy guarantees, allows for a smooth tradeoff between convergence rate and differential privacy levels. Numerical results demonstrate the advantages of FLORAS compared with the baseline AirComp method, and validate that our analytical results can guide the design of privacy-preserving FL with different tradeoff requirements on the model convergence and privacy levels.

[1]  Marten van Dijk,et al.  Generalizing DP-SGD with Shuffling and Batching Clipping , 2022, ArXiv.

[2]  Deniz Gündüz,et al.  Over-the-Air Federated Learning with Energy Harvesting Devices , 2022, GLOBECOM 2022 - 2022 IEEE Global Communications Conference.

[3]  Deniz Gündüz,et al.  Time-Correlated Sparsification for Efficient Over-the-Air Model Aggregation in Wireless Federated Learning , 2022, ICC 2022 - IEEE International Conference on Communications.

[4]  H. Poor,et al.  Random Orthogonalization for Federated Learning in Massive MIMO Systems , 2022, ICC 2022 - IEEE International Conference on Communications.

[5]  Siheng Chen,et al.  Dynamic Differential-Privacy Preserving SGD , 2021, ArXiv.

[6]  Rose Qingyang Hu,et al.  User Scheduling for Federated Learning Through Over-the-Air Computation , 2021, 2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall).

[7]  Yuan Liu,et al.  Differentially Private Federated Learning in Multi-Cell Networks , 2021, 2021 IEEE/CIC International Conference on Communications in China (ICCC).

[8]  Jinfeng Yi,et al.  Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy , 2021, ICML.

[9]  H. Vincent Poor,et al.  Low-Latency Federated Learning Over Wireless Channels With Differential Privacy , 2021, IEEE Journal on Selected Areas in Communications.

[10]  Mehdi Bennis,et al.  Joint Client Scheduling and Resource Allocation Under Channel Uncertainty in Federated Learning , 2021, IEEE Transactions on Communications.

[11]  Deniz Gündüz,et al.  Dynamic Scheduling for Over-the-Air Federated Edge Learning With Energy Constraints , 2021, IEEE Journal on Selected Areas in Communications.

[12]  K. Letaief,et al.  Convergence Analysis and System Design for Federated Learning Over Wireless Networks , 2021, IEEE Journal on Selected Areas in Communications.

[13]  Kaibin Huang,et al.  Deploying Federated Learning in Large-Scale Cellular Networks: Spatial Convergence Analysis , 2021, IEEE Transactions on Wireless Communications.

[14]  Jang-Won Lee,et al.  Adaptive Transmission Scheduling in Wireless Networks for Asynchronous Federated Learning , 2021, IEEE Journal on Selected Areas in Communications.

[15]  Alex Kulesza,et al.  Learning with User-Level Privacy , 2021, NeurIPS.

[16]  Onur Günlü,et al.  Federated Learning with Local Differential Privacy: Trade-Offs Between Privacy, Utility, and Communication , 2021, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]  Cong Shen,et al.  Design and Analysis of Uplink and Downlink Communications for Federated Learning , 2020, IEEE Journal on Selected Areas in Communications.

[18]  Deniz Gündüz,et al.  Private Wireless Federated Learning with Anonymous Over-the-Air Computation , 2020, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[19]  Deniz Gündüz,et al.  Blind Federated Edge Learning , 2020, IEEE Transactions on Wireless Communications.

[20]  Yonina C. Eldar,et al.  Over-the-Air Federated Learning From Heterogeneous Data , 2020, IEEE Transactions on Signal Processing.

[21]  Kaibin Huang,et al.  Over-the-Air Computing for Wireless Data Aggregation in Massive IoT , 2020, IEEE Wireless Communications.

[22]  Osvaldo Simeone,et al.  Privacy for Free: Wireless Federated Learning via Uncoded Transmission With Adaptive Power Control , 2020, IEEE Journal on Selected Areas in Communications.

[23]  Rui Hu,et al.  Personalized Federated Learning With Differential Privacy , 2020, IEEE Internet of Things Journal.

[24]  Jie Xu,et al.  Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspective , 2020, IEEE Transactions on Wireless Communications.

[25]  Ming Li,et al.  Wireless Federated Learning with Local Differential Privacy , 2020, 2020 IEEE International Symposium on Information Theory (ISIT).

[26]  H. Vincent Poor,et al.  Convergence Time Optimization for Federated Learning Over Wireless Networks , 2020, IEEE Transactions on Wireless Communications.

[27]  Kaibin Huang,et al.  Optimal Power Control for Over-the-Air Computation , 2019, 2019 IEEE Global Communications Conference (GLOBECOM).

[28]  H. Vincent Poor,et al.  Federated Learning With Differential Privacy: Algorithms and Performance Analysis , 2019, IEEE Transactions on Information Forensics and Security.

[29]  Tony Q. S. Quek,et al.  On Safeguarding Privacy and Security in the Framework of Federated Learning , 2019, IEEE Network.

[30]  Li Zhang,et al.  Rényi Differential Privacy of the Sampled Gaussian Mechanism , 2019, ArXiv.

[31]  Kobi Cohen,et al.  On Analog Gradient Descent Learning Over Multiple Access Fading Channels , 2019, IEEE Transactions on Signal Processing.

[32]  Solmaz Niknam,et al.  Federated Learning for Wireless Communications: Motivation, Opportunities, and Challenges , 2019, IEEE Communications Magazine.

[33]  Deniz Gündüz,et al.  Federated Learning Over Wireless Fading Channels , 2019, IEEE Transactions on Wireless Communications.

[34]  Xiang Li,et al.  On the Convergence of FedAvg on Non-IID Data , 2019, ICLR.

[35]  H. B. McMahan,et al.  Differentially Private Learning with Adaptive Clipping , 2019, NeurIPS.

[36]  Zhi Ding,et al.  Federated Learning via Over-the-Air Computation , 2018, IEEE Transactions on Wireless Communications.

[37]  Kaibin Huang,et al.  Broadband Analog Aggregation for Low-Latency Federated Edge Learning , 2018, IEEE Transactions on Wireless Communications.

[38]  Yang Song,et al.  Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning , 2018, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[39]  Peng Jiang,et al.  A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication , 2018, NeurIPS.

[40]  Marina Blanton,et al.  Secure Multiparty Computation , 2011, Encyclopedia of Cryptography and Security.

[41]  Borja Balle,et al.  Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences , 2018, NeurIPS.

[42]  Sanjiv Kumar,et al.  cpSGD: Communication-efficient and differentially-private distributed SGD , 2018, NeurIPS.

[43]  Sebastian U. Stich,et al.  Local SGD Converges Fast and Communicates Little , 2018, ICLR.

[44]  Ilya Mironov,et al.  Rényi Differential Privacy , 2017, 2017 IEEE 30th Computer Security Foundations Symposium (CSF).

[45]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[46]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[47]  Thomas Steinke,et al.  Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds , 2016, TCC.

[48]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[49]  Vitaly Shmatikov,et al.  Privacy-preserving deep learning , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[50]  Dimitris S. Papailiopoulos,et al.  Perturbed Iterate Analysis for Asynchronous Stochastic Optimization , 2015, SIAM J. Optim..

[51]  Xiao-Li Meng,et al.  An unexpected encounter with Cauchy and L\'evy , 2015, 1505.01957.

[52]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[53]  Peter Harremoës,et al.  Rényi Divergence and Kullback-Leibler Divergence , 2012, IEEE Transactions on Information Theory.

[54]  Michael Gastpar,et al.  Computation Over Multiple-Access Channels , 2007, IEEE Transactions on Information Theory.

[55]  Samuel Kotz,et al.  A truncated Cauchy distribution , 2006 .

[56]  David Tse,et al.  Fundamentals of Wireless Communication , 2005 .

[57]  Wenliang Du,et al.  Secure multi-party computation problems and their applications: a review and open problems , 2001, NSPW '01.

[58]  Whitfield Diffie,et al.  New Directions in Cryptography , 1976, IEEE Trans. Inf. Theory.

[59]  Stefania Sesia,et al.  LTE - The UMTS Long Term Evolution , 2009 .