Distributed Modelling Approaches for Data Privacy Preserving

Recently, machine learning has been developing rapidly. There is no doubt that data plays an important role in machine learning. However, it is hard to make full use of the data from a large amount of nodes to collaboratively train a good model with data privacy preserving. In this paper, we study and analyze several decentralized machine learning algorithms regarding to privacy protection, and propose a smart contract-based decentralized federated learning algorithm. We also propose a decentralized topology-based machine learning algorithm to solve the problems caused by star-topology network. Based on it, we further present a novel method of model aggregation based on distillation to break the conventional constrain of federated learning the models of different nodes shall have the same network structure. We also use several methods to generate synthetic dataset from raw dataset to train models with data privacy protected. Finally, we analyze and compare different distributed machine learning algorithms through the experiments.

[1]  Huchuan Lu,et al.  Deep Mutual Learning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  A. Krizhevsky.  Convolutional Deep Belief Networks on CIFAR-10 , 2010 .

[3]  Mauro Barni,et al.  Oblivious Neural Network Computing via Homomorphic Encryption , 2007, EURASIP J. Inf. Secur..

[4]  Aaron Roth,et al.  The Algorithmic Foundations of Differential Privacy , 2014, Found. Trends Theor. Comput. Sci..

[5]  Josh Benaloh,et al.  Secret Sharing Homomorphisms: Keeping Shares of A Secret Sharing , 1986, CRYPTO.

[6]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[7]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[8]  Rich Caruana,et al.  Model compression , 2006, KDD '06.

[9]  Peter Stone,et al.  A century-long commitment to assessing artificial intelligence and its impact on society , 2018, Commun. ACM.

[10]  Payman Mohassel,et al.  SecureML: A System for Scalable Privacy-Preserving Machine Learning , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[11]  Qing Ling,et al.  On the Convergence of Decentralized Gradient Descent , 2013, SIAM J. Optim..

[12]  David J. Wu,et al.  Using Homomorphic Encryption for Large Scale Statistical Analysis , 2012 .

[13]  Yao Lu,et al.  Oblivious Neural Network Predictions via MiniONN Transformations , 2017, IACR Cryptol. ePrint Arch..

[14]  Kin K. Leung,et al.  Adaptive Federated Learning in Resource Constrained Edge Computing Systems , 2018, IEEE Journal on Selected Areas in Communications.

[15]  Shiho Moriai,et al.  Privacy-Preserving Deep Learning via Additively Homomorphic Encryption , 2018, IEEE Transactions on Information Forensics and Security.

[16]  Wei Zhang,et al.  Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent , 2017, NIPS.

[17]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[18]  Jun Sakuma,et al.  Using Fully Homomorphic Encryption for Statistical Analysis of Categorical, Ordinal and Numerical Data , 2016, NDSS.

[19]  Sarvar Patel,et al.  Practical Secure Aggregation for Privacy-Preserving Machine Learning , 2017, IACR Cryptol. ePrint Arch..

[20]  Mike Rosulek.  Improvements for Gate-Hiding Garbled Circuits , 2017, INDOCRYPT.

[21]  Alexei A. Efros,et al.  Dataset Distillation , 2018, ArXiv.

[22]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[23]  Hongyi Zhang,et al.  mixup: Beyond Empirical Risk Minimization , 2017, ICLR.

[24]  Rui Zhang,et al.  A Hybrid Approach to Privacy-Preserving Federated Learning , 2018, Informatik Spektrum.

[25]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[26]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[27]  Chinmay Hegde,et al.  Collaborative Deep Learning in Fixed Topology Networks , 2017, NIPS.

[28]  Whitfield Diffie,et al.  New Directions in Cryptography , 1976, IEEE Trans. Inf. Theory.