Modeling program behaviors by hidden Markov models for intrusion detection

Intrusion detection is an important technique in the defense-in-depth network security framework and a hot topic in computer network security in recent years. In this paper, a new efficient intrusion detection method based on hidden Markov models (HMMs) is presented. HMMs are applied to model the normal program behaviors using traces of system calls issued by processes. The output probability of a sequence of system calls is calculated by the normal model built. If the probability of a sequence in a trace is below a certain threshold, the sequence is flagged as a mismatch. If the ratio between the mismatches and all the sequences in a trace exceeds another threshold, the trace is then considered as a possible intrusion. The method is implemented and tested on the sendmail system call data from the University of New Mexico. Experimental results show that the performance of the proposed method in intrusion detection is better than other methods.

[1]  Shigeki Goto,et al.  A new intrusion detection method based on discriminant analysis , 2001 .

[2]  Yehuda Vardi,et al.  A Hybrid High-Order Markov Chain Model for Computer Intrusion Detection , 2001 .

[3]  V. Rao Vemuri,et al.  Use of K-Nearest Neighbor classifier for intrusion detection , 2002, Comput. Secur..

[4]  Marc Dacier,et al.  Intrusion Detection Using Variable-Length Audit Trail Patterns , 2000, Recent Advances in Intrusion Detection.

[5]  Qiang Chen,et al.  Probabilistic techniques for intrusion detection based on computer audit data , 2001, IEEE Trans. Syst. Man Cybern. Part A.

[6]  S. E. Smaha.  Haystack: an intrusion detection system , 1988, [Proceedings 1988] Fourth Aerospace Computer Security Applications.

[7]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[8]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[9]  Sung-Bae Cho,et al.  Efficient anomaly detection by modeling privilege flows using hidden Markov model , 2003, Comput. Secur..

[10]  Connie M. Borror,et al.  Robustness of the Markov-chain model for cyber-attack detection , 2004, IEEE Transactions on Reliability.

[11]  Yiguo Qiao,et al.  Anomaly intrusion detection method based on HMM , 2002 .

[12]  Carla E. Brodley,et al.  Machine learning techniques for the computer security domain of anomaly detection , 2000 .

[13]  Barak A. Pearlmutter,et al.  Detecting intrusions using system calls: alternative data models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[14]  Zhongmin Cai,et al.  A rough set theory based method for anomaly intrusion detection in computer network systems , 2003, Expert Syst. J. Knowl. Eng..

[15]  V. Rao Vemuri,et al.  Robust Support Vector Machines for Anomaly Detection in Computer Security , 2003, ICMLA.

[16]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[17]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[18]  Alfonso Valdes,et al.  Next-generation Intrusion Detection Expert System (NIDES)A Summary , 1997 .

[19]  David G. Stork,et al.  Pattern Classification , 1973 .

[20]  Yuxin Ding,et al.  Host-based intrusion detection using dynamic and static behavioral models , 2003, Pattern Recognit..