Obfuscated Malicious Code Detection with Path Condition Analysis

Code obfuscation is one of the main methods to hide malicious code. This paper proposes a new dynamic method which can effectively detect obfuscated malicious code. This method uses ISR to conduct dynamic debugging. The constraint solving during debugging process can detect deeply hidden malicious code by covering different execution paths. Besides, for malicious code that reads external resources, usually the detection of abnormal behaviors can only be detected by taking the resources into consideration. The method in this paper has better accuracy by locating the external resources precisely and combining it with the analysis of original malicious code. According to the experiment result of some anti-virus software, our prototype system can obviously improve the detection efficiency.

[1]  Christopher Krügel,et al.  Shellzer: A Tool for the Dynamic Analysis of Malicious Shellcode , 2011, RAID.

[2]  Jonathon T. Giffin,et al.  Impeding Malware Analysis Using Conditional Code Obfuscation , 2008, NDSS.

[3]  Jules Desharnais,et al.  Static Detection of Malicious Code in Executable Programs , 2000 .

[4]  Fred Cohen,et al.  Computer viruses—theory and experiments , 1990 .

[5]  Christopher Krügel,et al.  Behavior-based Spyware Detection , 2006, USENIX Security Symposium.

[6]  Saumya K. Debray,et al.  Obfuscation of executable code to improve resistance to static disassembly , 2003, CCS '03.

[7]  Christopher Krügel,et al.  Static Disassembly of Obfuscated Binaries , 2004, USENIX Security Symposium.

[8]  Simin Nadjm-Tehrani,et al.  Crowdroid: behavior-based malware detection system for Android , 2011, SPSM '11.

[9]  Somesh Jha,et al.  Static Analysis of Executables to Detect Malicious Patterns , 2003, USENIX Security Symposium.

[10]  Niels Provos,et al.  SHELLOS: Enabling Fast Detection and Forensic Analysis of Code Injection Attacks , 2011, USENIX Security Symposium.

[11]  Christopher Krügel,et al.  Limits of Static Analysis for Malware Detection , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[12]  Saumya K. Debray,et al.  Deobfuscation: reverse engineering obfuscated code , 2005, 12th Working Conference on Reverse Engineering (WCRE'05).

[13]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[14]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[15]  Bezawada Bruhadeshwar,et al.  Signature Generation and Detection of Malware Families , 2008, ACISP.

[16]  Lorenzo Martignoni,et al.  A Fistful of Red-Pills: How to Automatically Generate Procedures to Detect CPU Emulators , 2009, WOOT.

[17]  Zhenkai Liang,et al.  BitBlaze: A New Approach to Computer Security via Binary Analysis , 2008, ICISS.

[18]  Evangelos P. Markatos,et al.  Emulation-Based Detection of Non-self-contained Polymorphic Shellcode , 2007, RAID.

[19]  Ed Skoudis,et al.  Malware: Fighting Malicious Code , 2003 .