Program monitoring based on automaton simulation

To check whether a program behaves in expectation, program monitoring systems are used for intrusion detection. This article presents a program monitoring system using automaton simulation based on the state graphs extracted from C programs through static analysis. For complete state graph construction, a pointer alias analysis method is proposed to solve the function pointers for obtaining actual control flows. After compiling, programs are instrumented with probes to report the internal states when they are running. A program monitor is built in the kernel of Linux system, which monitors the states of programs from probes and checks the paths of execution. This monitoring system could respond to the abnormal behaviors immediately to protect the systems and programs from further damages.

[1]  Qin Zhao,et al.  Transparent dynamic instrumentation , 2012, VEE '12.

[2]  Tal Garfinkel,et al.  A Virtual Machine Introspection Based Architecture for Intrusion Detection , 2003, NDSS.

[3]  Elchanan Mossel,et al.  Sorting and Selection in Posets , 2007, SIAM J. Comput..

[4]  David A. Wagner,et al.  Intrusion detection via static analysis , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[5]  Barbara G. Ryder,et al.  Precise Call Graphs for C Programs with Function Pointers , 2004, Automated Software Engineering.

[6]  Alain Deutsch,et al.  Interprocedural may-alias analysis for pointers: beyond k-limiting , 1994, PLDI '94.

[7]  A HofmeyrSteven,et al.  Intrusion Detection via System Call Traces , 1997 .

[8]  Pavan Kumar Chittimalli,et al.  GEMS: A Generic Model Based Source Code Instrumentation Framework , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[9]  Dilma Da Silva,et al.  Proceedings of the 8th ACM SIGPLAN/SIGOPS conference on Virtual Execution Environments , 2012, VEE 2012.

[10]  Susan Horwitz,et al.  Precise flow-insensitive may-alias analysis is NP-hard , 1997, TOPL.

[11]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[12]  Steven A. Hofmeyr,et al.  Intrusion Detection via System Call Traces , 1997, IEEE Softw..

[13]  Wen-mei W. Hwu,et al.  Modular interprocedural pointer analysis using access paths: design, implementation, and evaluation , 2000, PLDI '00.

[14]  Yan Chen,et al.  VNM: A Novel Method to Reduce the Overhead of Program Instrumentation , 2009, 2009 WRI World Congress on Software Engineering.

[15]  J. Michael Spivey,et al.  Fast, accurate call graph profiling , 2004, Softw. Pract. Exp..

[16]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[17]  Manuvir Das,et al.  Unification-based pointer analysis with directional assignments , 2000, PLDI '00.

[18]  Chris Hankin,et al.  Efficient field-sensitive pointer analysis of C , 2007, TOPL.