On Gray-Box Program Tracking for Anomaly Detection

Many host-based anomaly detection systems monitor a process ostensibly running a known program by observing the system calls the process makes. Numerous improvements to the precision of this approach have been proposed, such as tracking system call sequences, and various "gray-box" extensions such as examining the program counter or return addresses on the stack when system calls are made. In this paper, we perform the first systematic study of a wide spectrum of such methods. We show that prior approaches can be organized along three axes, revealing new possibilities for system-call-based program tracking. Through an empirical analysis of this design space, we shed light on the benefits and costs of various points in the space and identify new regions that appear to outperform prior approaches. In separate contributions, we demonstrate novel mimicry attacks on a recent proposal using return addresses for system-call-based program tracking, and then suggest randomization techniques to make such attacks more difficult.

[1]  Somesh Jha,et al.  Formalizing sensitivity in static analysis for intrusion detection , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[2]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[3]  David A. Wagner,et al.  Mimicry attacks on host-based intrusion detection systems , 2002, CCS '02.

[4]  Niels Provos,et al.  Preventing Privilege Escalation , 2003, USENIX Security Symposium.

[5]  David Aucsmith,et al.  Tamper Resistant Software: An Implementation , 1996, Information Hiding.

[6]  Flemming Nielson,et al.  Semantics with applications - a formal introduction , 1992, Wiley professional computing.

[7]  David H. Ackley,et al.  Randomized instruction set emulation to disrupt binary code injection attacks , 2003, CCS '03.

[8]  Derek Bruening,et al.  Secure Execution via Program Shepherding , 2002, USENIX Security Symposium.

[9]  David A. Wagner,et al.  Intrusion detection via static analysis , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[10]  Gregory R. Andrews,et al.  Disassembly of executable code revisited , 2002, Ninth Working Conference on Reverse Engineering, 2002. Proceedings..

[11]  Daniel C. DuVarney,et al.  Address Obfuscation: An Efficient Approach to Combat a Broad Range of Memory Error Exploits , 2003, USENIX Security Symposium.

[12]  Robert Wahbe,et al.  Efficient software-based fault isolation , 1994, SOSP '93.

[13]  Marc Dacier,et al.  An Intrusion-Detection System Based on the Teiresias Pattern- Discovery Algorithm , 1999 .

[14]  Niels Provos,et al.  Improving Host Security with System Call Policies , 2003, USENIX Security Symposium.

[15]  Kymie M. C. Tan,et al.  "Why 6?" Defining the operational limits of stide, an anomaly-based intrusion detector , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[16]  John McHugh,et al.  Hiding Intrusions: From the Abnormal to the Normal and Beyond , 2002, Information Hiding.

[17]  Weibo Gong,et al.  Anomaly detection using call stack information , 2003, 2003 Symposium on Security and Privacy, 2003..

[18]  Tzi-cker Chiueh,et al.  A Binary Rewriting Defense Against Stack based Buffer Overflow Attacks , 2003, USENIX Annual Technical Conference, General Track.

[19]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[20]  Angelos D. Keromytis,et al.  Countering code-injection attacks with instruction-set randomization , 2003, CCS '03.

[21]  Marc Dacier,et al.  Intrusion Detection Using Variable-Length Audit Trail Patterns , 2000, Recent Advances in Intrusion Detection.

[22]  David Wagner,et al.  Janus: an Approach for Confinement of Untrusted Applications , 1999 .

[23]  Somesh Jha,et al.  Detecting Manipulated Remote Call Streams , 2002, USENIX Security Symposium.

[24]  Alec Wolman,et al.  Instrumentation and optimization of Win32/intel executables using Etch , 1997 .

[25]  Daniel C. DuVarney,et al.  Model-carrying code: a practical approach for safe execution of untrusted applications , 2003, SOSP '03.

[26]  Somesh Jha,et al.  Efficient Context-Sensitive Intrusion Detection , 2004, NDSS.

[27]  Aris Floratos,et al.  Combinatorial pattern discovery in biological sequences: The TEIRESIAS algorithm [published erratum appears in Bioinformatics 1998;14(2): 229] , 1998, Bioinform..