Privacy-preserving Decentralized Federated Learning

In this paper, we develop SecureD-FL, a privacy-preserving decentralized federated learning algorithm, i.e., without the traditional centralized aggregation server. For the decentralized aggregation, we employ the Alternating Direction Method of Multiplier (ADMM) and examine its privacy weakness. To address the privacy risk, we introduce a communication pattern inspired by the combinatorial block design theory and establish its theoretical privacy guarantee. We also propose an efficient algorithm to construct such a communication pattern. We evaluate our method on image classification and next-word prediction applications over federated benchmark datasets with nine and fifteen distributed sites hosting training data. While preserving privacy, SecureD-FL performs comparably to the standard centralized federated learning method; the degradation in test accuracy is only up to 0.73%.

[1]  Rui Hu,et al.  DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy , 2018, IEEE Transactions on Information Forensics and Security.

[2]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[3]  Carlo Fischione,et al.  On the Privacy of Optimization , 2017 .

[4]  Vitaly Shmatikov,et al.  Membership Inference Attacks Against Machine Learning Models , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[5]  Vitaly Shmatikov,et al.  Privacy-preserving deep learning , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[6]  Hubert Eichner,et al.  Federated Learning for Mobile Keyboard Prediction , 2018, ArXiv.

[7]  Asaf Ferber,et al.  Almost all Steiner triple systems are almost resolvable , 2019, Forum of Mathematics, Sigma.

[8]  Oded Goldreich,et al.  Foundations of Cryptography: Volume 2, Basic Applications , 2004 .

[9]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[10]  Ivan Damgård,et al.  Multiparty Computation from Somewhat Homomorphic Encryption , 2012, IACR Cryptol. ePrint Arch..

[11]  Junda Liu,et al.  Multi-enterprise networking , 2000 .

[12]  Hubert Eichner,et al.  APPLIED FEDERATED LEARNING: IMPROVING GOOGLE KEYBOARD QUERY SUGGESTIONS , 2018, ArXiv.

[13]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[14]  D. K. Ray-Chaudhuri,et al.  Solution of Kirkman''s schoolgirl problem , 1971 .

[15]  Mariana Raykova,et al.  Secure Computation for Machine Learning With SPDZ , 2019, ArXiv.

[16]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[17]  Pascal Paillier,et al.  Public-Key Cryptosystems Based on Composite Degree Residuosity Classes , 1999, EUROCRYPT.

[18]  Sarvar Patel,et al.  Practical Secure Aggregation for Privacy-Preserving Machine Learning , 2017, IACR Cryptol. ePrint Arch..

[19]  Rui Zhang,et al.  A Hybrid Approach to Privacy-Preserving Federated Learning , 2018, Informatik Spektrum.

[20]  C. Colbourn,et al.  The CRC handbook of combinatorial designs , edited by Charles J. Colbourn and Jeffrey H. Dinitz. Pp. 784. $89.95. 1996. ISBN 0-8493-8948-8 (CRC). , 1997, The Mathematical Gazette.

[21]  Swaroop Ramaswamy,et al.  Federated Learning for Emoji Prediction in a Mobile Keyboard , 2019, ArXiv.

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Tom Ouyang,et al.  Federated Learning Of Out-Of-Vocabulary Words , 2019, ArXiv.

[24]  Pascal Brisset,et al.  Solving the Kirkman's Schoolgirl Problem in a Few Seconds , 2002, CP.

[25]  Amit Sahai,et al.  Multi-Input Functional Encryption , 2014, IACR Cryptol. ePrint Arch..

[26]  Qing Ling,et al.  Decentralized Dynamic Optimization Through the Alternating Direction Method of Multipliers , 2013, IEEE Transactions on Signal Processing.

[27]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Qing Ling,et al.  Weighted ADMM for Fast Decentralized Network Optimization , 2016, IEEE Transactions on Signal Processing.

[29]  Peter Keevash.  HYPERGRAPH MATCHINGS AND DESIGNS , 2018, Proceedings of the International Congress of Mathematicians (ICM 2018).

[30]  Pascal Brisset,et al.  Solving Kirkman’s Schoolgirl Problem in a Few Seconds , 2004, Constraints.

[31]  Eric W. Weisstein.  Social Golfer Problem , 2005 .

[32]  Gregory Cohen,et al.  EMNIST: an extension of MNIST to handwritten letters , 2017, CVPR 2017.

[33]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[34]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[35]  Sebastian Caldas,et al.  LEAF: A Benchmark for Federated Settings , 2018, ArXiv.

[36]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[38]  Cynthia Dwork,et al.  Differential Privacy , 2006, ICALP.

[39]  Mingyan Liu,et al.  Improving the Privacy and Accuracy of ADMM-Based Distributed Algorithms , 2018, ICML.

[40]  Douglas R. Stinson,et al.  Combinatorial designs: constructions and analysis , 2003, SIGA.

[41]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[42]  Amir Houmansadr,et al.  Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[43]  Daniel Rueckert,et al.  A generic framework for privacy preserving deep learning , 2018, ArXiv.

[44]  Matthew Kwan.  Almost all Steiner triple systems have perfect matchings , 2016, Proceedings of the London Mathematical Society.

[45]  Matt J. Kusner,et al.  QUOTIENT: Two-Party Secure Neural Network Training and Prediction , 2019, CCS.

[46]  Charles C. Lindner,et al.  Kirkman Triple Systems , 2008 .

[47]  Xiaoqiang Ren,et al.  Secure and privacy preserving average consensus , 2017, 2017 11th Asian Control Conference (ASCC).

[48]  Yongqiang Wang,et al.  ADMM Based Privacy-Preserving Decentralized Optimization , 2017, IEEE Transactions on Information Forensics and Security.

[49]  Vitaly Shmatikov,et al.  Exploiting Unintended Feature Leakage in Collaborative Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[50]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[51]  Wotao Yin,et al.  Global Convergence of ADMM in Nonconvex Nonsmooth Optimization , 2015, Journal of Scientific Computing.

[52]  William Shakespeare,et al.  Complete Works of William Shakespeare , 1854 .

[53]  Runhua Xu,et al.  HybridAlpha: An Efficient Approach for Privacy-Preserving Federated Learning , 2019, AISec@CCS.