Application of deep reinforcement learning to intrusion detection for supervised problems

Abstract The application of new techniques to increase the performance of intrusion detection systems is crucial in modern data networks with a growing threat of cyber-attacks. These attacks impose a greater risk on network services that are increasingly important from a social end economical point of view. In this work we present a novel application of several deep reinforcement learning (DRL) algorithms to intrusion detection using a labeled dataset. We present how to perform supervised learning based on a DRL framework. The implementation of a reward function aligned with the detection of intrusions is extremely difficult for Intrusion Detection Systems (IDS) since there is no automatic way to identify intrusions. Usually the identification is performed manually and stored in datasets of network features associated with intrusion events. These datasets are used to train supervised machine learning algorithms for classifying intrusion events. In this paper we apply DRL using two of these datasets: NSL-KDD and AWID datasets. As a novel approach, we have made a conceptual modification of the classic DRL paradigm (based on interaction with a live environment), replacing the environment with a sampling function of recorded training intrusions. This new pseudo-environment, in addition to sampling the training dataset, generates rewards based on detection errors found during training. We present the results of applying our technique to four of the most relevant DRL models: Deep Q-Network (DQN), Double Deep Q-Network (DDQN), Policy Gradient (PG) and Actor-Critic (AC). The best results are obtained for the DDQN algorithm. We show that DRL, with our model and some parameter adjustments, can improve the results of intrusion detection in comparison with current machine learning techniques. Besides, the classifier obtained with DRL is faster than alternative models. A comprehensive comparison of the results obtained with other machine learning models is provided for the AWID and NSL-KDD datasets, together with the lessons learned from the application of several design alternatives to the four DRL models.

[1]  Qi Shi,et al.  A Deep Learning Approach to Network Intrusion Detection , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[2]  Andreas Hotho,et al.  A Survey of Network-based Intrusion Detection Data Sets , 2019, Comput. Secur..

[3]  Xianbin Wang,et al.  Machine learning techniques for intrusion detection on public dataset , 2016, 2016 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE).

[4]  Wei-Yang Lin,et al.  Intrusion detection by machine learning: A review , 2009, Expert Syst. Appl..

[5]  Ali A. Ghorbani,et al.  A detailed analysis of the KDD CUP 99 data set , 2009, 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications.

[6]  Shahin Ara Begum,et al.  A Comparison of Intrusion Detection by K-Means and Fuzzy C-Means Clustering Algorithm Over the NSL-KDD Dataset , 2017, 2017 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC).

[7]  S. M. Kovalev,et al.  Advanced Temporal-Difference Learning for Intrusion Detection , 2015 .

[8]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[9]  Manuel López Martín,et al.  Adversarial environment reinforcement learning algorithm for intrusion detection , 2019, Comput. Networks.

[10]  Xin Xu,et al.  A Kernel-Based Reinforcement Learning Approach to Dynamic Behavior Modeling of Intrusion Detection , 2007, ISNN.

[11]  Sean Carlisto de Alvarenga,et al.  A survey of intrusion detection in Internet of Things , 2017, J. Netw. Comput. Appl..

[12]  Mansoor Alam,et al.  A Deep Learning Approach for Network Intrusion Detection System , 2016, EAI Endorsed Trans. Security Safety.

[13]  Bo Li,et al.  Attack Detection for Wireless Enterprise Network: a Machine Learning Approach , 2018, 2018 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC).

[14]  Xin Wang,et al.  Machine Learning for Networking: Workflow, Advances and Opportunities , 2017, IEEE Network.

[15]  Bu-Sung Lee,et al.  Autoencoder-based network anomaly detection , 2018, 2018 Wireless Telecommunications Symposium (WTS).

[16]  João Paulo Papa,et al.  Internet of Things: A survey on machine learning-based intrusion detection approaches , 2019, Comput. Networks.

[17]  Jianye Hao,et al.  Falsification of Cyber-Physical Systems Using Deep Reinforcement Learning , 2018, IEEE Transactions on Software Engineering.

[18]  Robert Babuska,et al.  A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[19]  Daniel S. Berman,et al.  A Survey of Deep Learning Methods for Cyber Security , 2019, Inf..

[20]  Xin Xu,et al.  A Sparse Kernel-Based Least-Squares Temporal Difference Algorithm for Reinforcement Learning , 2006, ICNC.

[21]  Bart De Schutter,et al.  Multi-agent Reinforcement Learning: An Overview , 2010 .

[22]  Georgios Kambourakis,et al.  Intrusion Detection in 802.11 Networks: Empirical Evaluation of Threats and a Public Dataset , 2016, IEEE Communications Surveys & Tutorials.

[23]  Deepa Pavithran,et al.  A Survey of Intrusion Detection Models based on NSL-KDD Data Set , 2018, 2018 Fifth HCT Information Technology Trends (ITT).

[24]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[25]  Bhagyashree Deokar,et al.  Intrusion Detection System using Log Files and Reinforcement Learning , 2012 .

[26]  Murat Aydos,et al.  A review on cyber security datasets for machine learning algorithms , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[27]  Hao Xu,et al.  Deep reinforecement learning based optimal defense for cyber-physical system in presence of unknown cyber-attack , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[28]  Jaime Lloret,et al.  Shallow neural network with kernel approximation for prediction problems in highly demanding data networks , 2019, Expert Syst. Appl..

[29]  Daniel Kudenko,et al.  Multi-agent Reinforcement Learning for Intrusion Detection , 2007, Adaptive Agents and Multi-Agents Systems.

[30]  Miad Faezipour,et al.  Effective Features Selection and Machine Learning Classifiers for Improved Wireless Intrusion Detection , 2018, 2018 International Symposium on Networks, Computers and Communications (ISNCC).

[31]  Kwangjo Kim,et al.  Deep learning in intrusion detection perspective: Overview and further challenges , 2017, 2017 International Workshop on Big Data and Information Security (IWBIS).

[32]  Paul D. Yoo,et al.  From Intrusion Detection to Attacker Attribution: A Comprehensive Survey of Unsupervised Methods , 2018, IEEE Communications Surveys & Tutorials.

[33]  Jugal K. Kalita,et al.  Network Anomaly Detection: Methods, Systems and Tools , 2014, IEEE Communications Surveys & Tutorials.

[34]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[35]  Raouf Boutaba,et al.  A comprehensive survey on machine learning for networking: evolution, applications and research opportunities , 2018, Journal of Internet Services and Applications.

[36]  Young-June Choi,et al.  Performance Enhancement of Deep Neural Network Using Feature Selection and Preprocessing for Intrusion Detection , 2019, 2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC).

[37]  Zhongjiang Yan,et al.  Intrusion Detection for WiFi Network: A Deep Learning Approach , 2018, WICON.

[38]  Aboubaker Lasebae,et al.  An efficient deep learning model for intrusion classification and prediction in 5G and IoT networks , 2019, 2019 53rd Annual Conference on Information Sciences and Systems (CISS).