Dynamical System Theory for the Detection of Anomalous Behavior in Computer Programs

Code injection is a common approach which is utilized to exploit applications. We introduce some of the well-established techniques and formalisms of dynamical system theory into analysis of program behavior via system calls to detect code injections into an applications execution space. We accept a program as a blackbox dynamical system whose internals are not known, but whose output we can observe. The blackbox system observable in our model is the system calls the program makes. The collected system calls are treated as signals which are used to reconstruct the system's phase space. Then, by using the well-established techniques from dynamical system theory, we quantify the amount of complexity of the system's (program's) behavior. The change in the behavior of a compromised system is detected as anomalous behavior compared with the baseline established from a clean program. We test the proposed approach against DARPA-98 dataset and a real-world exploit and present code injection experiments to show the applicability of our approach.

[1]  Christopher Krügel,et al.  Anomalous system call detection , 2006, TSEC.

[2]  D. L. Hudson,et al.  Applying continuous chaotic modeling to cardiac signal analysis , 1996 .

[3]  Ali A. Ghorbani,et al.  IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS 1 Toward Credible Evaluation of Anomaly-Based Intrusion-Detection Methods , 2022 .

[4]  Steven A. Hofmeyr,et al.  Intrusion Detection via System Call Traces , 1997, IEEE Softw..

[5]  Srini Ramaswamy,et al.  A Dynamical System Approach to Intrusion Detection Using System Call Analysis , 2007, SEKE.

[6]  C L Webber,et al.  Dynamical assessment of physiological systems and states using recurrence plot strategies. , 1994, Journal of applied physiology.

[7]  S. Sharma,et al.  An exploratory study of chaos in human-Machine system dynamics , 2006, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[8]  M. Small.  Applied Nonlinear Time Series Analysis: Applications in Physics, Physiology and Finance , 2005 .

[9]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[10]  Metin Akay,et al.  Influence of the vagus nerve on respiratory patterns during early maturation , 2005, IEEE Transactions on Biomedical Engineering.

[11]  A HofmeyrSteven,et al.  Intrusion Detection via System Call Traces , 1997 .

[12]  R. Sekar,et al.  Dataflow anomaly detection , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[13]  Salvatore J. Stolfo,et al.  Learning Rules from System Call Arguments and Sequences for Anomaly 20 Detection , 2003 .

[14]  S. Strogatz.  Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering , 1995 .

[15]  D. Ruelle,et al.  Recurrence Plots of Dynamical Systems , 1987 .

[16]  Song Li,et al.  Temporal signatures for intrusion detection , 2001, Seventeenth Annual Computer Security Applications Conference.

[17]  R.K. Cunningham,et al.  Evaluating intrusion detection systems: the 1998 DARPA off-line intrusion detection evaluation , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[18]  Geoffrey H. Kuenning,et al.  Detecting insider threats by monitoring system call activity , 2003, IEEE Systems, Man and Cybernetics SocietyInformation Assurance Workshop, 2003..

[19]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[20]  B. LeBaron,et al.  Nonlinear Dynamics, Chaos, and Instability: Statistical Theory and Economic Evidence , 1991 .

[21]  Yiguo Qiao,et al.  Anomaly intrusion detection method based on HMM , 2002 .

[22]  Raman K. Mehra,et al.  Detection and classification of intrusions and faults using sequences of system calls , 2001, SGMD.

[23]  A HofmeyrSteven,et al.  Intrusion detection using sequences of system calls , 1998 .

[24]  S M Pincus,et al.  Approximate entropy as a measure of system complexity. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[25]  LewisLundy,et al.  Detection and classification of intrusions and faults using sequences of system calls , 2001 .

[26]  H. Kantz,et al.  Nonlinear time series analysis , 1997 .

[27]  Floris Takens,et al.  On the numerical determination of the dimension of an attractor , 1985 .