A Bray-Curtis Weighted Automaton for Detecting Malicious Code Through System-Call Analysis

Malicious code detection is one of the top subjects of interest for intrusion detection systems in today's computer security research areas. In this paper we propose a new heuristic method for detecting malicious code through system call matching, which also takes in consideration the time of the system call, by using an adaptive search for an extended Aho-Corasick automaton supporting a subset of the regular expressions language, through the use of a normalization technique known as the Bray-Curtis (Sorensen) distance. We will also discuss how this technique can be applied to enrich the set of existing rules from the knowledge base for improving the detection rate.

[1]  Andrew H. Sung,et al.  Static analyzer of vicious executables (SAVE) , 2004, 20th Annual Computer Security Applications Conference.

[2]  Nasir Memon,et al.  EFFICIENT STATIC ANALYSIS OF EXECUTABLES FOR DETECTING MALICIOUS BEHAVIORS , 2005 .

[3]  Fred Cohen,et al.  Computer viruses—theory and experiments , 1990 .

[4]  Alfred V. Aho,et al.  Efficient string matching , 1975, Commun. ACM.

[5]  Alfonso Valdes,et al.  Next-generation Intrusion Detection Expert System (NIDES)A Summary , 1997 .

[6]  J. J. Whelan,et al.  5th international conference on software engineering , 1981, SOEN.

[7]  Mark Weiser,et al.  Program Slicing , 1981, IEEE Transactions on Software Engineering.

[8]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[9]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[10]  Steve R. White,et al.  An Undetectable Computer Virus , 2000 .

[11]  Andrew Walenstein,et al.  Statistical signatures for fast filtering of instruction-substituting metamorphic malware , 2007, WORM '07.

[12]  Tibor Gyimóthy,et al.  Interprocedural static slicing of binary executables , 2003, Proceedings Third IEEE International Workshop on Source Code Analysis and Manipulation.

[14]  Peter G. Neumann,et al.  EMERALD: Event Monitoring Enabling Responses to Anomalous Live Disturbances , 1997, CCS 2002.

[15]  Weibo Gong,et al.  Anomaly detection using call stack information , 2003, 2003 Symposium on Security and Privacy, 2003..

[16]  William Landi,et al.  Undecidability of static analysis , 1992, LOPL.

[17]  Cristina Cifuentes,et al.  Intraprocedural static slicing of binary executables , 1997, 1997 Proceedings International Conference on Software Maintenance.