A Game-Theoretic Approach to Design Secure and Resilient Distributed Support Vector Machines

Distributed support vector machines (DSVMs) have been developed to solve large-scale classification problems in networked systems with a large number of sensors and control units. However, the systems become more vulnerable, as detection and defense are increasingly difficult and expensive. This paper aims to develop secure and resilient DSVM algorithms under adversarial environments in which an attacker can manipulate the training data to achieve his objective. We establish a game-theoretic framework to capture the conflicting interests between an adversary and a set of distributed data processing units. The Nash equilibrium of the game allows predicting the outcome of learning algorithms in adversarial environments and enhancing the resilience of the machine learning through dynamic distributed learning algorithms. We prove that the convergence of the distributed algorithm is guaranteed without assumptions on the training data or network topologies. Numerical experiments are conducted to corroborate the results. We show that the network topology plays an important role in the security of DSVM. Networks with fewer nodes and higher average degrees are more secure. Moreover, a balanced network is found to be less vulnerable to attacks.

[1]  Jeannette M. Wing,et al.  Game strategies in network security , 2005, International Journal of Information Security.

[2]  Yi Yang,et al.  Semantic Pooling for Complex Event Analysis in Untrimmed Videos , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Guang-Bin Huang,et al.  Extreme Learning Machine for Multilayer Perceptron , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Ivor W. Tsang,et al.  Domain Transfer Multiple Kernel Learning , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Yi Yang,et al.  Image Classification by Cross-Media Active Learning With Privileged Information , 2016, IEEE Transactions on Multimedia.

[6]  T. Başar,et al.  Dynamic Noncooperative Game Theory , 1982 .

[7]  Ross J. Anderson.  Why information security is hard - an economic perspective , 2001, Seventeenth Annual Computer Security Applications Conference.

[8]  Ivor W. Tsang,et al.  Learning With Augmented Features for Supervised and Semi-Supervised Heterogeneous Domain Adaptation , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Panagiotis Tsakalides,et al.  Training a SVM-based classifier in distributed sensor networks , 2006, 2006 14th European Signal Processing Conference.

[10]  Y. Radhika,et al.  Atmospheric Temperature Prediction using Support Vector Machines , 2009 .

[11]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[12]  Quanyan Zhu,et al.  Game-Theoretic Approach to Feedback-Driven Multi-stage Moving Target Defense , 2013, GameSec.

[13]  Tobias Scheffer,et al.  Stackelberg games for adversarial prediction problems , 2011, KDD.

[14]  H. Nikaidô.  On von Neumann’s minimax theorem , 1954 .

[15]  Ivor W. Tsang,et al.  Visual Event Recognition in Videos by Learning from Web Data , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Sushil Jajodia,et al.  Toward measuring network security using attack graphs , 2007, QoP '07.

[17]  Chris Clifton,et al.  Classifier evaluation and attribute selection against active adversaries , 2010, Data Mining and Knowledge Discovery.

[18]  Quanyan Zhu,et al.  Consensus-based transfer linear support vector machines for decentralized multi-task multi-agent learning , 2018, 2018 52nd Annual Conference on Information Sciences and Systems (CISS).

[19]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[20]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[21]  Quanyan Zhu,et al.  Game theory meets network security and privacy , 2013, CSUR.

[22]  François Poulet,et al.  Classifying one billion data with a new distributed svm algorithm , 2006, 2006 International Conference onResearch, Innovation and Vision for the Future.

[23]  Blaine Nelson,et al.  The security of machine learning , 2010, Machine Learning.

[24]  Blaine Nelson,et al.  Misleading Learners: Co-opting Your Spam Filter , 2009 .

[25]  Rui Zhang,et al.  A game-theoretic analysis of label flipping attacks on distributed support vector machines , 2017, 2017 51st Annual Conference on Information Sciences and Systems (CISS).

[26]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[27]  Quanyan Zhu,et al.  A game-theoretic defense against data poisoning attacks in distributed support vector machines , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[28]  Muharram Mansoorizadeh,et al.  Multi-Level Fuzzy Min-Max Neural Network Classifier , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Ivor W. Tsang,et al.  Core Vector Machines: Fast SVM Training on Very Large Data Sets , 2005, J. Mach. Learn. Res..

[30]  Shie Mannor,et al.  Robustness and Regularization of Support Vector Machines , 2008, J. Mach. Learn. Res..

[31]  Quanyan Zhu,et al.  A Dual Perturbation Approach for Differential Private ADMM-Based Distributed Empirical Risk Minimization , 2016, AISec@CCS.

[32]  Rui Zhang,et al.  Secure and resilient distributed machine learning under adversarial environments , 2015, 2015 18th International Conference on Information Fusion (Fusion).

[33]  M. Verleysen,et al.  Classification in the Presence of Label Noise: A Survey , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[34]  T. Kavitha,et al.  Security Vulnerabilities In Wireless Sensor Networks: A Survey , 2010 .

[35]  D. Sculley,et al.  Relaxed online SVMs for spam filtering , 2007, SIGIR.

[36]  Quanyan Zhu,et al.  A Bi-Level Game Approach to Attack-Aware Cyber Insurance of Computer Networks , 2017, IEEE Journal on Selected Areas in Communications.

[37]  Francisco Herrera,et al.  A unifying view on dataset shift in classification , 2012, Pattern Recognit..

[38]  Fabio Roli,et al.  Randomized Prediction Games for Adversarial Machine Learning , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Quanyan Zhu,et al.  Dynamic Differential Privacy for ADMM-Based Distributed Classification Learning , 2017, IEEE Transactions on Information Forensics and Security.

[40]  Claude E. Shannon,et al.  Communication theory of secrecy systems , 1949, Bell Syst. Tech. J..

[41]  Ivor W. Tsang,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1 Domain Adaptation from Multiple Sources: A Domain- , 2022 .

[42]  Tobias Scheffer,et al.  Nash Equilibria of Static Prediction Games , 2009, NIPS.

[43]  Sébastien Marcel,et al.  Spoofing in 2D face recognition with 3D masks and anti-spoofing with Kinect , 2013, 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS).

[44]  Xiaojin Zhu,et al.  Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners , 2015, AAAI.

[45]  Pavel Laskov,et al.  Practical Evasion of a Learning-Based Classifier: A Case Study , 2014, 2014 IEEE Symposium on Security and Privacy.

[46]  Quanyan Zhu,et al.  Heterogeneous learning in zero-sum stochastic games with incomplete information , 2011, 49th IEEE Conference on Decision and Control (CDC).

[47]  Ling Shao,et al.  Learning Deep and Wide: A Spectral Method for Learning Deep Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Francisco Herrera,et al.  Study on the Impact of Partition-Induced Dataset Shift on $k$-Fold Cross-Validation , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[49]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[50]  Quanyan Zhu,et al.  Game-Theoretic Methods for Robustness, Security, and Resilience of Cyberphysical Control Systems: Games-in-Games Principle for Optimal Cross-Layer Resilient Control Systems , 2015, IEEE Control Systems.

[51]  Pietro Michiardi,et al.  Game theoretic analysis of security in mobile ad hoc networks , 2002 .

[52]  Christos-Savvas Bouganis,et al.  Novel Cascade FPGA Accelerator for Support Vector Machines Classification , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[53]  Pedro M. Domingos,et al.  Adversarial classification , 2004, KDD.

[54]  Quanyan Zhu,et al.  Distributed strategic learning with application to network security , 2011, Proceedings of the 2011 American Control Conference.

[55]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..