CodedPaddedFL and CodedSecAgg: Straggler Mitigation and Secure Aggregation in Federated Learning

We present two novel coded federated learning (FL) schemes for linear regression that mitigate the effect of straggling devices. The first scheme, CodedPaddedFL, mitigates the effect of straggling devices while retaining the privacy level of conventional FL. Particularly, it combines one-time padding for user data privacy with gradient codes to yield resiliency against straggling devices. To apply one-time padding to real data, our scheme exploits a fixed-point arithmetic representation of the data. For a scenario with 25 devices, CodedPaddedFL achieves a speed-up factor of 6.6 and 9.2 for an accuracy of 95% and 85% on the MMIST and Fashion-MNIST datasets, respectively, compared to conventional FL. Furthermore, it yields similar performance in terms of latency compared to a recently proposed scheme by Prakash et al. without the shortcoming of additional leakage of private data. The second scheme, CodedSecAgg, provides straggler resiliency and robustness against model inversion attacks and is based on Shamir’s secret sharing. CodedSecAgg outperforms state-of-the-art secure aggregation schemes such as LightSecAgg by a speed-up factor of 6.6–14.6, depending on the number of colluding devices, on the MNIST dataset for a scenario with 120 devices, at the expense of a 30% increase in latency compared to CodedPaddedFL.

[1]  Anit Kumar Sahu,et al.  Federated Optimization in Heterogeneous Networks , 2018, MLSys.

[2]  Octavian Catrina,et al.  Secure Computation with Fixed-Point Numbers , 2010, Financial Cryptography.

[3]  Amir Salman Avestimehr,et al.  LightSecAgg: Rethinking Secure Aggregation in Federated Learning , 2021, ArXiv.

[4]  Kannan Ramchandran,et al.  Speeding Up Distributed Machine Learning Using Codes , 2015, IEEE Transactions on Information Theory.

[5]  Farzin Haddadpour,et al.  On the optimal recovery threshold of coded matrix multiplication , 2017, 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[6]  Adi Shamir,et al.  How to share a secret , 1979, CACM.

[7]  A. Salman Avestimehr,et al.  Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning , 2020, IEEE Journal on Selected Areas in Information Theory.

[8]  Alexandre Graell i Amat,et al.  Rateless Codes for Low-Latency Distributed Inference in Mobile Edge Computing , 2021, ArXiv.

[9]  Mohammad Ali Maddah-Ali,et al.  Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix Multiplication , 2017, NIPS.

[10]  Qinghua Liu,et al.  Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization , 2020, NeurIPS.

[11]  Alexandros G. Dimakis,et al.  Gradient Coding: Avoiding Stragglers in Distributed Learning , 2017, ICML.

[12]  Shusen Yang,et al.  Asynchronous Federated Learning with Differential Privacy for Edge Intelligence , 2019, ArXiv.

[13]  Osvaldo Simeone,et al.  On Model Coding for Distributed Inference and Transmission in Mobile Edge Computing Systems , 2019, IEEE Communications Letters.

[14]  Tancrède Lepoint,et al.  Secure Single-Server Aggregation with (Poly)Logarithmic Overhead , 2020, IACR Cryptol. ePrint Arch..

[15]  Pulkit Grover,et al.  “Short-Dot”: Computing Large Linear Transforms Distributedly Using Coded Short Dot Products , 2017, IEEE Transactions on Information Theory.

[16]  Claude E. Shannon,et al.  Communication theory of secrecy systems , 1949, Bell Syst. Tech. J..

[17]  Suhas N. Diggavi,et al.  Straggler Mitigation in Distributed Optimization Through Data Encoding , 2017, NIPS.

[18]  Sarvar Patel,et al.  Practical Secure Aggregation for Privacy-Preserving Machine Learning , 2017, IACR Cryptol. ePrint Arch..

[19]  Yang Song,et al.  Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning , 2018, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[20]  Alexandre Graell i Amat,et al.  Private Edge Computing for Linear Inference Based on Secret Sharing , 2020, GLOBECOM 2020 - 2020 IEEE Global Communications Conference.

[21]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[22]  Jakub Konecný,et al.  On the Outsized Importance of Learning Rates in Local Update Methods , 2020, ArXiv.

[23]  Anit Kumar Sahu,et al.  Federated Learning: Challenges, Methods, and Future Directions , 2019, IEEE Signal Processing Magazine.

[24]  A. Elkordy,et al.  Secure Aggregation with Heterogeneous Quantization in Federated Learning , 2020, IEEE Transactions on Communications.

[25]  Amir Salman Avestimehr,et al.  Coded computation over heterogeneous clusters , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[26]  Kannan Ramchandran,et al.  FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning , 2020, ArXiv.

[27]  Indranil Gupta,et al.  Asynchronous Federated Optimization , 2019, ArXiv.

[28]  Stephen A. Jarvis,et al.  SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low Overhead , 2019, IEEE Transactions on Computers.

[29]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[30]  Albin Severinson,et al.  Block-Diagonal and LT Codes for Distributed Computing With Straggling Servers , 2017, IEEE Transactions on Communications.

[31]  Kshitiz Malik,et al.  Federated Learning with Buffered Asynchronous Aggregation , 2021, ArXiv.

[32]  Yizhou Zhao,et al.  Information Theoretic Secure Aggregation with User Dropouts , 2021, 2021 IEEE International Symposium on Information Theory (ISIT).

[33]  Nageen Himayat,et al.  Coded Computing for Low-Latency Federated Learning Over Wireless Edge Networks , 2020, IEEE Journal on Selected Areas in Communications.

[34]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[35]  George J. Pappas,et al.  Achieving Linear Convergence in Federated Learning under Objective and Systems Heterogeneity , 2021, ArXiv.

[36]  Somesh Jha,et al.  Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures , 2015, CCS.

[37]  Amir Salman Avestimehr,et al.  Secure Aggregation for Buffered Asynchronous Federated Learning , 2021, ArXiv.

[38]  Mohammad Ali Maddah-Ali,et al.  A Unified Coding Framework for Distributed Computing with Straggling Servers , 2016, 2016 IEEE Globecom Workshops (GC Wkshps).