Long-Span Program Behavior Modeling and Attack Detection

Intertwined developments between program attacks and defenses witness the evolution of program anomaly detection methods. Emerging categories of program attacks, e.g., non-control data attacks and data-oriented programming, are able to comply with normal trace patterns at local views. This article points out the deficiency of existing program anomaly detection models against new attacks and presents long-span behavior anomaly detection (LAD), a model based on mildly context-sensitive grammar verification. The key feature of LAD is its reasoning of correlations among arbitrary events that occurred in long program traces. It extends existing correlation analysis between events at a stack snapshot, e.g., paired call and ret, to correlation analysis among events that historically occurred during the execution. The proposed method leverages specialized machine learning techniques to probe normal program behavior boundaries in vast high-dimensional detection space. Its two-stage modeling/detection design analyzes event correlation at both binary and quantitative levels. Our prototype successfully detects all reproduced real-world attacks against sshd, libpcre, and sendmail. The detection procedure incurs 0.1 ms to 1.3 ms overhead to profile and analyze a single behavior instance that consists of tens of thousands of function call or system call events.

[1]  Debin Gao,et al.  Behavioral Distance Measurement Using Hidden Markov Models , 2006, RAID.

[2]  Guofei Gu,et al.  Using an Ensemble of One-Class SVM Classifiers to Harden Payload-based Anomaly Detection Systems , 2006, Sixth International Conference on Data Mining (ICDM'06).

[3]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[4]  Stefano Zanero,et al.  Selecting and Improving System Call Models for Anomaly Detection , 2009, DIMVA.

[5]  Stefano Zanero.  Behavioral Intrusion Detection , 2004, ISCIS.

[6]  Ali Abbasi,et al.  A gray-box DPDA-based intrusion detection technique using system-call monitoring , 2011, CEAS '11.

[7]  Ramarathnam Venkatesan,et al.  Pattern Mining for Future Attacks , 2009 .

[8]  Geoffrey K. Pullum,et al.  Context-Freeness and the Computer Processing of Human Languages , 1983, ACL.

[9]  Zhenkai Liang,et al.  Data-Oriented Programming: On the Expressiveness of Non-control Data Attacks , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[10]  Santosh Biswas,et al.  Sequencegram: n-gram modeling of system calls for program based anomaly detection , 2011, 2011 Third International Conference on Communication Systems and Networks (COMSNETS 2011).

[11]  Zhen Liu,et al.  Combining static analysis and dynamic learning to build accurate intrusion detection models , 2005, Third IEEE International Workshop on Information Assurance (IWIA'05).

[12]  Somesh Jha,et al.  Synthesizing Near-Optimal Malware Specifications from Suspicious Behaviors , 2010, 2010 IEEE Symposium on Security and Privacy.

[13]  Somesh Jha,et al.  Formalizing sensitivity in static analysis for intrusion detection , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[14]  Stefano Zanero,et al.  Detecting Intrusions through System Call Sequence and Argument Analysis , 2010, IEEE Transactions on Dependable and Secure Computing.

[15]  David A. Wagner,et al.  Intrusion detection via static analysis , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[16]  Steven A. Hofmeyr,et al.  Intrusion Detection via System Call Traces , 1997, IEEE Softw..

[17]  Weibo Gong,et al.  Anomaly detection using call stack information , 2003, 2003 Symposium on Security and Privacy, 2003..

[18]  Nong Ye,et al.  A Markov Chain Model of Temporal Behavior for Anomaly Detection , 2000 .

[19]  Jun Xu,et al.  Non-Control-Data Attacks Are Realistic Threats , 2005, USENIX Security Symposium.

[20]  Dorothy E. Denning,et al.  An Intrusion-Detection Model , 1987, IEEE Transactions on Software Engineering.

[21]  Christopher Krügel,et al.  Comprehensive approach to intrusion detection alert correlation , 2004, IEEE Transactions on Dependable and Secure Computing.

[22]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[23]  Vinod Yegneswaran,et al.  BotHunter: Detecting Malware Infection Through IDS-Driven Dialog Correlation , 2007, USENIX Security Symposium.

[24]  Barbara G. Ryder,et al.  A Sharper Sense of Self: Probabilistic Reasoning of Program Behaviors for Anomaly Detection with Context Sensitivity , 2016, 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[25]  Naren Ramakrishnan,et al.  Unearthing Stealthy Program Attacks Buried in Extremely Long Execution Paths , 2015, CCS.

[26]  Barak A. Pearlmutter,et al.  Detecting intrusions using system calls: alternative data models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[27]  Somesh Jha,et al.  Efficient Context-Sensitive Intrusion Detection , 2004, NDSS.

[28]  Wenke Lee,et al.  McPAD: A multiple classifier system for accurate payload-based anomaly detection , 2009, Comput. Networks.

[29]  Md. Saiful Islam,et al.  Detecting Unknown Anomalous Program Behavior Using API System Calls , 2011 .

[30]  Anup K. Ghosh,et al.  Detecting anomalous and unknown intrusions against programs , 1998, Proceedings 14th Annual Computer Security Applications Conference (Cat. No.98EX217).

[31]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[32]  David A. Wagner,et al.  Mimicry attacks on host-based intrusion detection systems , 2002, CCS '02.

[33]  Danfeng Yao,et al.  Program Anomaly Detection: Methodology and Practices , 2016, CCS.

[34]  Luo Si,et al.  LEAPS: Detecting Camouflaged Attacks with Statistical Learning Guided by Program Analysis , 2015, 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks.

[35]  Barbara G. Ryder,et al.  Probabilistic Program Modeling for High-Precision Anomaly Classification , 2015, 2015 IEEE 28th Computer Security Foundations Symposium.

[36]  Stefan Savage,et al.  Inferring Internet denial-of-service activity , 2001, TOCS.

[37]  Trent Jaeger,et al.  GRIFFIN: Guarding Control Flows Using Intel Processor Trace , 2017, ASPLOS.

[38]  Stephanie Forrest,et al.  The Evolution of System-Call Monitoring , 2008, 2008 Annual Computer Security Applications Conference (ACSAC).

[39]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[40]  Wenke Lee,et al.  Understanding precision in host based intrusion detection: formal analysis and practical models , 2007 .

[41]  Hovav Shacham,et al.  On the effectiveness of address-space randomization , 2004, CCS '04.

[42]  Naren Ramakrishnan,et al.  Detection of stealthy malware activities with traffic causality and scalable triggering relation discovery , 2014, AsiaCCS.

[43]  Arun Kejariwal,et al.  Trin-Trin: Who’s Calling? A Pin-Based Dynamic Call Graph Extraction Framework , 2012, International Journal of Parallel Programming.

[44]  George Candea,et al.  Failure sketching: a technique for automated root cause diagnosis of in-production failures , 2015, SOSP.

[45]  Bernhard Schölkopf,et al.  Support Vector Method for Novelty Detection , 1999, NIPS.

[46]  Christopher Krügel,et al.  A quantitative study of accuracy in system call-based malware detection , 2012, ISSTA 2012.

[47]  James Newsome,et al.  Paragraph: Thwarting Signature Learning by Training Maliciously , 2006, RAID.

[48]  Naren Ramakrishnan,et al.  Causality reasoning about network events for detecting stealthy malware activities , 2016, Comput. Secur..

[49]  J. Doug Tygar,et al.  Adversarial machine learning , 2019, AISec '11.

[50]  Anil Somayaji,et al.  Lookahead Pairs and Full Sequences : A Tale of Two Anomaly Detection Methods , 2007 .

[51]  John McHugh,et al.  An Anthropological Approach to Studying CSIRTs , 2014, IEEE Security & Privacy.

[52]  Giovanni Vigna,et al.  Swaddler: An Approach for the Anomaly-Based Detection of State Violations in Web Applications , 2007, RAID.

[53]  Nick Feamster,et al.  Behavioral Clustering of HTTP-Based Malware and Signature Generation Using Malicious Network Traces , 2010, NSDI.

[54]  David Brumley,et al.  RICH: Automatically Protecting Against Integer-Based Vulnerabilities , 2007, NDSS.

[55]  Somesh Jha,et al.  Environment-Sensitive Intrusion Detection , 2005, RAID.

[56]  Barbara G. Ryder,et al.  A Formal Framework for Program Anomaly Detection , 2015, RAID.

[57]  Jonathon T. Giffin,et al.  Understanding Precision in Host Based Intrusion Detection , 2007, RAID.

[58]  Christopher Krügel,et al.  Scalable, Behavior-Based Malware Clustering , 2009, NDSS.

[59]  Jan Vitek,et al.  Efficient intrusion detection using automaton inlining , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[60]  J. Sukarno Mertoguno,et al.  Human Decision Making Model for Autonomic Cyber Systems , 2014, Int. J. Artif. Intell. Tools.

[61]  R. Sekar,et al.  Dataflow anomaly detection , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[62]  Zhenkai Liang,et al.  Automatic Generation of Data-Oriented Exploits , 2015, USENIX Security Symposium.