Generalization of Deep Learning for Cyber-Physical System Security: A Survey

Cyber-Physical Systems (CPSs)have become ubiquitous in recent years and has become the core of modern critical infrastructure and industrial applications. Therefore, ensuring security is a prime concern. Due to the success of Deep Learning (DL)in a multitude of domains, development of DL based CPS security applications have received increased interest in the past few years. Developing generalized models is critical since the models have to perform well under threats that they havent trained on. However, despite the broad body of work on using DL for ensuring the security of CPSs, to our best knowledge very little work exists where the focus is on the generalization capabilities of these DL applications. In this paper, we intend to provide a concise survey of the regularization methods for DL algorithms used in security-related applications in CPSs and thus could be used to improve the generalization capability of DL based cyber-physical system based security applications. Further, we provide a brief insight into the current challenges and future directions as well.

[1]  Daniel L. Marino,et al.  An Adversarial Approach for Explainable AI in Intrusion Detection Systems , 2018, IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society.

[2]  Milos Manic,et al.  Parallalizable deep self-organizing maps for image classification , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[3]  Yuancheng Li,et al.  A Hybrid Malicious Code Detection Method based on Deep Learning , 2015 .

[4]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[5]  Geoffrey E. Hinton,et al.  Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[6]  David Gunning,et al.  DARPA's explainable artificial intelligence (XAI) program , 2019, IUI.

[7]  Samy Bengio,et al.  Understanding deep learning requires rethinking generalization , 2016, ICLR.

[8]  Wei Wang,et al.  Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network , 2018, Journal of Ambient Intelligence and Humanized Computing.

[9]  Peter König,et al.  Data augmentation instead of explicit regularization , 2018, ArXiv.

[10]  Milos Manic,et al.  Toward Explainable Deep Neural Network Based Anomaly Detection , 2018, 2018 11th International Conference on Human System Interaction (HSI).

[11]  Insup Lee,et al.  Cyber-physical systems: The next computing revolution , 2010, Design Automation Conference.

[12]  Quoc V. Le,et al.  Measuring Invariances in Deep Networks , 2009, NIPS.

[13]  Khulumani Sibanda,et al.  Training Set Size for Generalization Ability of Artificial Neural Networks in Forecasting TCP/IP Traffic Trends , 2015 .

[14]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[15]  Siddharth Sridhar,et al.  Cyber–Physical System Security for the Electric Power Grid , 2012, Proceedings of the IEEE.

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[18]  Claudia Eckert,et al.  Deep Learning for Classification of Malware System Call Sequences , 2016, Australasian Conference on Artificial Intelligence.

[19]  Christoph H. Lampert,et al.  Data-Dependent Stability of Stochastic Gradient Descent , 2017, ICML.

[20]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Surya Ganguli,et al.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization , 2014, NIPS.

[22]  Mohd Faizal Abdollah,et al.  Analysis of Features Selection and Machine Learning Classifier in Android Malware Detection , 2014, 2014 International Conference on Information Science & Applications (ICISA).

[23]  Klaus-Robert Müller,et al.  Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models , 2017, ArXiv.

[24]  K. V. Prema,et al.  Generalization Capability of Artificial Neural Network Incorporated with Pruning Method , 2011, ADCONS.

[25]  Shingo Mabu,et al.  Enhancing the generalization ability of neural networks through controlling the hidden layers , 2009, Appl. Soft Comput..

[26]  Yoram Singer,et al.  Train faster, generalize better: Stability of stochastic gradient descent , 2015, ICML.

[27]  Léon Bottou,et al.  Stochastic Gradient Descent Tricks , 2012, Neural Networks: Tricks of the Trade.

[28]  Serge Andrianov,et al.  Comparison of Regularization Methods for ImageNet Classification with Deep Convolutional Neural Networks , 2014 .

[29]  Marc'Aurelio Ranzato,et al.  Large Scale Distributed Deep Networks , 2012, NIPS.

[30]  Samy Bengio,et al.  Adversarial examples in the physical world , 2016, ICLR.

[31]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[32]  Yann LeCun,et al.  Deep learning with Elastic Averaging SGD , 2014, NIPS.

[33]  Erhan Guven,et al.  A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection , 2016, IEEE Communications Surveys & Tutorials.

[34]  Furong Huang,et al.  Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition , 2015, COLT.

[35]  Ananthram Swami,et al.  The Limitations of Deep Learning in Adversarial Settings , 2015, 2016 IEEE European Symposium on Security and Privacy (EuroS&P).

[36]  Adnan Anwar,et al.  Cyber Security of Smart Grid Infrastructure , 2014, ArXiv.

[37]  Sushanta Karmakar,et al.  A Neural Network based system for Intrusion Detection and attack classification , 2016, 2016 Twenty Second National Conference on Communication (NCC).

[38]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[39]  Je-Won Kang,et al.  Intrusion Detection System Using Deep Neural Network for In-Vehicle Network Security , 2016, PloS one.

[40]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Jorge Nocedal,et al.  On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima , 2016, ICLR.

[42]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[43]  Leslie Pack Kaelbling,et al.  Generalization in Deep Learning , 2017, ArXiv.

[44]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[45]  Ling Gao,et al.  An Intrusion Detection Model Based on Deep Belief Networks , 2014 .

[46]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[47]  Sridhar Adepu,et al.  Anomaly Detection in Cyber Physical Systems Using Recurrent Neural Networks , 2017, 2017 IEEE 18th International Symposium on High Assurance Systems Engineering (HASE).

[48]  Pradeep Dubey,et al.  Distributed Deep Learning Using Synchronous Stochastic Gradient Descent , 2016, ArXiv.

[49]  Daniel L. Marino,et al.  Interpretable Data-Driven Modeling in Biomass Preprocessing , 2018, 2018 11th International Conference on Human System Interaction (HSI).

[50]  Nathan Srebro,et al.  Exploring Generalization in Deep Learning , 2017, NIPS.

[51]  Anders Krogh,et al.  A Simple Weight Decay Can Improve Generalization , 1991, NIPS.

[52]  Jinoh Kim,et al.  A survey of deep learning-based network anomaly detection , 2017, Cluster Computing.

[53]  Yanfang Ye,et al.  DL 4 MD : A Deep Learning Framework for Intelligent Malware Detection , 2016 .

[54]  Taghi M. Khoshgoftaar,et al.  Deep learning applications and challenges in big data analytics , 2015, Journal of Big Data.

[55]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[56]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..