A Kernel-Based Reinforcement Learning Approach to Dynamic Behavior Modeling of Intrusion Detection

As an important active defense technique for computer networks, intrusion detection has received lots of attention in recent years. However, the performance of current intrusion detection systems (IDSs) is far from being satisfactory due to the increasing number of complex sequential attacks. Aiming at the above problem, in this paper, a novel kernel-based reinforcement learning method for sequential behavior modeling in host-based IDSs is proposed. Based on Markov process modeling of host-based intrusion detection using sequences of system calls, the performance optimization of IDSs is transformed to a sequential prediction problem using evaluative reward signals. By using the kernel-based learning prediction algorithm, i.e., the kernel least-squares temporal-difference (kernel LS-TD) algorithm, which implements LS-TD learning in a kernel-induced feature space, the nonlinear modeling and prediction problem for sequential behaviors in IDSs is efficiently solved. Experiments on system call data from the University of New Mexico illustrate that the proposed kernel-based RL approach can achieve better detection accuracy than previous sequential behavior modeling methods including Hidden Markov Models (HMMs) and linear TD algorithms.

[1]  Anup K. Ghosh,et al.  A Study in Using Neural Networks for Anomaly and Misuse Detection , 1999, USENIX Security Symposium.

[2]  Justin A. Boyan,et al.  Technical Update: Least-Squares Temporal Difference Learning , 2002, Machine Learning.

[3]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[4]  Yuxin Ding,et al.  Host-based intrusion detection using dynamic and static behavioral models , 2003, Pattern Recognit..

[5]  Xiao-Ping Zhang,et al.  Advances in Intelligent Computing, International Conference on Intelligent Computing, ICIC 2005, Hefei, China, August 23-26, 2005, Proceedings, Part I , 2005, ICIC.

[6]  Yew-Soon Ong,et al.  Advances in Natural Computation, First International Conference, ICNC 2005, Changsha, China, August 27-29, 2005, Proceedings, Part I , 2005, ICNC.

[7]  Xin Xu,et al.  A Reinforcement Learning Approach for Host-Based Intrusion Detection Using Sequences of System Calls , 2005, ICIC.

[8]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[9]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[10]  Risto Miikkulainen,et al.  Intrusion Detection with Neural Networks , 1997, NIPS.

[11]  Shie Mannor,et al.  The kernel recursive least-squares algorithm , 2004, IEEE Transactions on Signal Processing.

[12]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[13]  Xin Xu,et al.  A Sparse Kernel-Based Least-Squares Temporal Difference Algorithm for Reinforcement Learning , 2006, ICNC.

[14]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[15]  Susan M. Bridges,et al.  Mining fuzzy association rules and fuzzy frequency episodes for intrusion detection , 2000 .

[16]  Anupam Joshi,et al.  Fuzzy clustering for intrusion detection , 2003, The 12th IEEE International Conference on Fuzzy Systems, 2003. FUZZ '03..

[17]  Connie M. Borror,et al.  Robustness of the Markov-chain model for cyber-attack detection , 2004, IEEE Transactions on Reliability.

[18]  Sushil Jajodia,et al.  Detecting Novel Network Intrusions Using Bayes Estimators , 2001, SDM.