Tutorial: An Overview of Malware Detection and Evasion Techniques

This tutorial presents and motivates various malware detection tools and illustrates their usage on a clear example. We demonstrate how statically-extracted syntactic signatures can be used for quickly detecting simple variants of malware. Since such signatures can easily be obfuscated, we also present dynamically-extracted behavioral signatures which are obtained by running the malware in an isolated environment known as a sandbox. However, some malware can use sandbox detection to detect that they run in such an environment and so avoid exhibiting their malicious behavior. To counteract sandbox detection, we present concolic execution that can explore several paths of a binary. We conclude by showing how opaque predicates and JIT can be used to hinder concolic execution.

[1]  Christian S. Collberg,et al.  Distributed application tamper detection via continuous software updates , 2012, ACSAC '12.

[2]  Josephine Micallef,et al.  Detection of global, metamorphic malware variants using control and data flow analysis , 2012, MILCOM 2012 - 2012 IEEE Military Communications Conference.

[3]  Andrew Honig,et al.  Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software , 2012 .

[4]  Aditya P. Mathur,et al.  A Survey of Malware Detection Techniques , 2007 .

[5]  Paul A. Watters,et al.  Zero-day Malware Detection based on Supervised Learning Algorithms of API call Signatures , 2011, AusDM.

[6]  Saumya Debray,et al.  Symbolic Execution of Obfuscated Code , 2015, CCS.

[7]  Somesh Jha,et al.  A semantics-based approach to malware detection , 2007, POPL '07.

[8]  Christopher Krügel,et al.  A quantitative study of accuracy in system call-based malware detection , 2012, ISSTA 2012.

[9]  Christopher Krügel,et al.  BareCloud: Bare-metal Analysis-based Evasive Malware Detection , 2014, USENIX Security Symposium.

[10]  Aaron Walters,et al.  The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory , 2014 .

[11]  Christopher Krügel,et al.  Limits of Static Analysis for Malware Detection , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[12]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[13]  Christopher Krügel,et al.  SOK: (State of) The Art of War: Offensive Techniques in Binary Analysis , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[14]  Sanjay Kumar Sahay,et al.  Evolution and Detection of Polymorphic and Metamorphic Malwares: A Survey , 2014, ArXiv.

[15]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[16]  Somesh Jha,et al.  Semantics-aware malware detection , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[17]  Robert P. Goldberg,et al.  Survey of virtual machine research , 1974, Computer.

[18]  Vladimir A. Zakharov,et al.  On the Concept of Software Obfuscation in Computer Security , 2007, ISC.

[19]  Peter Szor,et al.  The Art of Computer Virus Research and Defense , 2005 .

[20]  Mansour Ahmadi,et al.  Semantic Malware Detection by Deploying Graph Mining , 2012 .