Machine Learning with Distributed Processing using Secure Divided Data: Towards Privacy-Preserving Advanced AI Processing in a Super-Smart Society

Towards the realization of a super-smart society, AI analysis methods that preserve the privacy of big data in cyberspace are being developed. From the viewpoint of developing machine learning as a secure and safe AI analysis method for users, many studies have been conducted in this field on 1) secure multiparty computation (SMC), 2) quasi-homomorphic encryption, and 3) federated learning, among other techniques. Previous studies have shown that both security and utility are essential for machine learning using confidential data. However, there is a trade-off between these two properties, and there are no known methods that satisfy both simultaneously at a high level. In this paper, as a superior method in both privacy-preserving of data and utility, we propose a learning method based on distributed processing using simple, secure, divided data and parameters. In this method, individual data and parameters are divided into multiple pieces using random numbers in advance, and each piece is stored in each server. The learning of the proposed method is achieved by using these data and parameters as they are divided and by repeating partial computations on each server and integrated computations at the central server. The advantages of the proposed method are the preservation of data privacy by not restoring the data and parameters during learning; the improvement of usability by realizing a machine learning method based on distributed processing, as federated learning does; and almost no degradation in accuracy compared to conventional methods. Based on the proposed method, we propose backpropagation and neural gas (NG) algorithms as examples of supervised and unsupervised machine learning applications. Our numerical simulation shows that these algorithms can achieve accuracy comparable to conventional models.

[1]  Zheli Liu,et al.  Secure Aggregation is Insecure: Category Inference Attack on Federated Learning , 2023, IEEE Transactions on Dependable and Secure Computing.

[2]  Toktam Mahmoodi,et al.  A survey: Distributed Machine Learning for 5G and beyond , 2022, Comput. Networks.

[3]  Yuanman Li,et al.  Privacy-preserving and verifiable deep learning inference based on secret sharing , 2022, Neurocomputing.

[4]  Xuefei Yin,et al.  A Comprehensive Survey of Privacy-preserving Federated Learning , 2021, ACM Comput. Surv..

[5]  Liangli Zhen,et al.  Partially Encrypted Multi-Party Computation for Federated Learning , 2021, 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid).

[6]  Yang Qin,et al.  A Selective Model Aggregation Approach in Federated Learning for Online Anomaly Detection , 2020, 2020 International Conferences on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics).

[7]  Muhammad Usman Asad,et al.  FedOpt: Towards Communication Efficiency and Privacy Preservation in Federated Learning , 2020, Applied Sciences.

[8]  Masatoshi Yoshikawa,et al.  FedSel: Federated SGD under Local Differential Privacy with Top-k Dimension Selection , 2020, DASFAA.

[9]  T. Robinson,et al.  Sustainable Development Goals , 2016, World Social Report.

[10]  H. Vincent Poor,et al.  Federated Learning With Differential Privacy: Algorithms and Performance Analysis , 2019, IEEE Transactions on Information Forensics and Security.

[11]  Hiromi Miyajima,et al.  Fast and Secure Back-Propagation Learning Using Vertically Partitioned Data with IoT , 2019, 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW).

[12]  Vladimir Kolesnikov,et al.  A Pragmatic Introduction to Secure Multi-Party Computation , 2019, Found. Trends Priv. Secur..

[13]  Rui Zhang,et al.  A Hybrid Approach to Privacy-Preserving Federated Learning , 2018, Informatik Spektrum.

[14]  Hao Deng,et al.  LoAdaBoost: Loss-Based AdaBoost Federated Machine Learning on medical Data , 2018, ArXiv.

[15]  Hubert Eichner,et al.  Federated Learning for Mobile Keyboard Prediction , 2018, ArXiv.

[16]  Jian Shen,et al.  Privacy preserving multi-party computation delegation for deep learning in cloud computing , 2018, Inf. Sci..

[17]  Walid Saad,et al.  Federated Learning for Ultra-Reliable Low-Latency V2V Communications , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[18]  Georg Carle,et al.  A management framework for secure multiparty computation in dynamic environments , 2018, NOMS 2018 - 2018 IEEE/IFIP Network Operations and Management Symposium.

[19]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[20]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[21]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[22]  Norio Shiratori,et al.  New Methods to Ensure Security to Increase User's Sense of Safety in Cloud Services , 2014, 2014 IEEE 11th Intl Conf on Ubiquitous Intelligence and Computing and 2014 IEEE 11th Intl Conf on Autonomic and Trusted Computing and 2014 IEEE 14th Intl Conf on Scalable Computing and Communications and Its Associated Workshops.

[23]  W. Stallings.  cryptography and network security principles and practices -5/E. , 2014 .

[24]  Amos Beimel,et al.  Secret-Sharing Schemes: A Survey , 2011, IWCC.

[25]  Anand D. Sarwate,et al.  Differentially Private Empirical Risk Minimization , 2009, J. Mach. Learn. Res..

[26]  Sheng Zhong,et al.  Privacy-Preserving Backpropagation Neural Network Learning , 2009, IEEE Transactions on Neural Networks.

[27]  Craig Gentry,et al.  Fully homomorphic encryption using ideal lattices , 2009, STOC '09.

[28]  Philip S. Yu,et al.  Privacy-Preserving Data Mining: Models and Algorithms , 2008 .

[29]  Ueli Maurer,et al.  General Secure Multi-party Computation from any Linear Secret-Sharing Scheme , 2000, EUROCRYPT.

[30]  Moni Naor,et al.  Adaptively secure multi-party computation , 1996, STOC '96.

[31]  Thomas Martinetz,et al.  'Neural-gas' network for vector quantization and its application to time-series prediction , 1993, IEEE Trans. Neural Networks.

[32]  Adi Shamir,et al.  How to share a secret , 1979, CACM.

[33]  Yuanxiong Guo,et al.  Concentrated Differentially Private Federated Learning With Performance Analysis , 2021, IEEE Open Journal of the Computer Society.

[34]  N. Shiratori,et al.  New Privacy Preserving Back Propagation Learning for Secure Multiparty Computation , 2016 .

[35]  Un Desa.  Transforming our world : The 2030 Agenda for Sustainable Development , 2016 .

[36]  Daniela Fischer,et al.  Cryptography And Network Security Principles And Practice , 2016 .

[37]  Dr. T. Karthikeyan,et al.  Survey on Recent Algorithms for Privacy Preserving Data mining , 2015 .

[38]  Shucheng Yu,et al.  Privacy Preserving Back-Propagation Neural Network Learning Made Practical with Cloud Computing , 2014, IEEE Transactions on Parallel and Distributed Systems.

[39]  Ashley Hammerbacher.  UNITED NATIONS FOUNDATION , 2012 .

[40]  Nico Schlitter,et al.  A Protocol for Privacy Preserving Neural Network Learning on Horizontally Partitioned Data , 2008 .

[41]  L. Fagerlund.  The United Nations , 1993 .