PsycoTrace: Virtual and Transparent Monitoring of a Process Self

PsycoTrace is a set of tools to protect a process $P$ from attacks that alter $P$ self as specified by its source code. $P$ self is specified in terms of legal traces of system calls and of assertions on $P$ status paired with each call. In turn, legal traces are specified through a context-free grammar returned by a static analysis of $P$ program that may also compute assertions. At run-time, each time $P$ invokes a system call, PsycoTrace checks that the trace is coherent with the grammar and assertions are satisfied. To increase overall robustness, PsycoTrace's run-time tool relies on two virtual machines that run, respectively, $P$ and the monitoring system. This strongly separates the monitored machine that runs $P$ from the monitoring one. The current implementation is fully transparent to $P$ but not to the OS because a kernel module in the monitored machine intercepts system calls.We describe PsycoTrace overall architecture and focus on the run-time and introspection tools that enable the monitoring machine to check that a trace is legal and to transparently access the memory of the other machine to evaluate assertions. Lastly, a preliminary evaluation of the run-time overhead is discussed.

[1]  Jun Xu,et al.  Non-Control-Data Attacks Are Realistic Threats , 2005, USENIX Security Symposium.

[2]  David A. Wagner,et al.  Mimicry attacks on host-based intrusion detection systems , 2002, CCS '02.

[3]  Tzi-cker Chiueh,et al.  Accurate and Automated System Call Policy-Based Intrusion Prevention , 2006, International Conference on Dependable Systems and Networks (DSN'06).

[4]  David A. Wagner,et al.  Intrusion detection via static analysis , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[5]  Somesh Jha,et al.  Detecting Manipulated Remote Call Streams , 2002, USENIX Security Symposium.

[6]  James R. Larus,et al.  Mining specifications , 2002, POPL '02.

[7]  Sandeep Bhatkar,et al.  Improving Attack Detection in Host-Based IDS by Learning Properties of System Call Arguments , 2005 .

[8]  Stephen McCamant,et al.  The Daikon system for dynamic detection of likely invariants , 2007, Sci. Comput. Program..

[9]  Tal Garfinkel,et al.  A Virtual Machine Introspection Based Architecture for Intrusion Detection , 2003, NDSS.

[10]  Andrew Warfield,et al.  Xen and the art of virtualization , 2003, SOSP '03.

[11]  Somesh Jha,et al.  Efficient Context-Sensitive Intrusion Detection , 2004, NDSS.

[12]  Somesh Jha,et al.  Formalizing sensitivity in static analysis for intrusion detection , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[13]  Massimo Bernaschi,et al.  Operating system enhancements to prevent the misuse of system calls , 2000, CCS.

[14]  Weibo Gong,et al.  Anomaly detection using call stack information , 2003, 2003 Symposium on Security and Privacy, 2003..