Classifying malicious system behavior using event propagation trees

Behavior-based analysis of dynamically executed software has become an established technique to identifying and analyzing potential malware. Most solutions rely on API or system call patterns to determine whether a sample is exhibiting malicious activity. Analysis is usually performed on demand and offers little insight into the current system state. In addition, the fixed nature of behavioral patterns is known to cause false-positives whenever a certain, potentially malicious action is used in a benign context. To combat these shortcomings, this paper proposes an analysis system capable of building event propagation trees from real-time kernel monitoring data. Distance-based anomaly detection is then used to find and highlight activities deviating from a predefined baseline established through heuristic clustering. The system was tested on a set of real-world data collected by a number of host-based agents distributed across a corporate network.

[1]  Robert Luh,et al.  Malicious Behavior Patterns , 2014, 2014 IEEE 8th International Symposium on Service Oriented System Engineering.

[2]  Felix C. Freiling,et al.  Visual analysis of malware behavior using treemaps and thread graphs , 2009, 2009 6th International Workshop on Visualization for Cyber Security.

[3]  R. Sekar,et al.  Dataflow anomaly detection , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[4]  E. LESTER SMITH,et al.  AND OTHERS , 2005 .

[5]  Carsten Willems,et al.  A Malware Instruction Set for Behavior-Based Analysis , 2010, Sicherheit.

[6]  Leonid Peshkin,et al.  Structure induction by lossless graph compression , 2007, 2007 Data Compression Conference (DCC'07).

[7]  Vlado Keselj,et al.  Detection of New Malicious Code Using N-grams Signatures , 2004, PST.

[8]  Lars Michael Kristensen,et al.  Coloured Petri Nets - Modelling and Validation of Concurrent Systems , 2009 .

[9]  Radu State,et al.  Malware analysis with graph kernels and support vector machines , 2009, 2009 4th International Conference on Malicious and Unwanted Software (MALWARE).

[10]  Christopher Krügel,et al.  A survey on automated dynamic malware-analysis techniques and tools , 2012, CSUR.

[11]  Alexander Pretschner,et al.  DAVAST: data-centric system level activity visualization , 2014, VizSec '14.

[12]  Carsten Willems,et al.  Automatic analysis of malware behavior using machine learning , 2011, J. Comput. Secur..

[13]  Yoseba K. Penya,et al.  Automatic Behaviour-based Analysis and Classification System for Malware Detection , 2010, ICEIS.

[14]  Mario Jino,et al.  Behavioral analysis of malicious code through network traffic and system call monitoring , 2011, Defense + Commercial Sensing.

[15]  Somesh Jha,et al.  Semantics-aware malware detection , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[16]  Christopher Krügel,et al.  Limits of Static Analysis for Malware Detection , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[17]  Somesh Jha,et al.  Mining specifications of malicious behavior , 2008, ISEC '08.

[18]  Jan van den Berg,et al.  Systems for Detecting Advanced Persistent Threats: A Development Roadmap Using Intelligent Data Analysis , 2012, 2012 International Conference on Cyber Security.

[19]  Ian H. Witten,et al.  Data mining - practical machine learning tools and techniques, Second Edition , 2005, The Morgan Kaufmann series in data management systems.

[20]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[21]  Luca Faust,et al.  Modern Operating Systems , 2016 .

[22]  Mark Heydenrych,et al.  An adaptive multi-agent architecture for critical information infrastructure protection , 2014 .

[23]  Victor A. Skormin,et al.  Using Behavioral Modeling and Customized Normalcy Profiles as Protection against Targeted Cyber-Attacks , 2012, MMM-ACNS.

[24]  Daniel A. Keim,et al.  A Survey of Visualization Systems for Malware Analysis , 2015, EuroVis.

[25]  Aditya P. Mathur,et al.  A Survey of Malware Detection Techniques , 2007 .

[26]  Felix C. Freiling,et al.  Toward Automated Dynamic Malware Analysis Using CWSandbox , 2007, IEEE Secur. Priv..

[27]  Heejo Lee,et al.  BinGraph: Discovering mutant malware using hierarchical semantic signatures , 2012, 2012 7th International Conference on Malicious and Unwanted Software.

[28]  Kris Kendall,et al.  Practical Malware Analysis , 2012, Netw. Secur..

[29]  Xiaoqi Jia,et al.  A Behavior Feature Generation Method for Obfuscated Malware Detection , 2012, 2012 International Conference on Computer Science and Service System.

[30]  Jiankun Hu,et al.  A Semantic Approach to Host-Based Intrusion Detection Systems Using Contiguousand Discontiguous System Call Patterns , 2014, IEEE Transactions on Computers.

[31]  Krzysztof Z. Gajos,et al.  Evaluation of Filesystem Provenance Visualization Tools , 2013, IEEE Transactions on Visualization and Computer Graphics.