Linear Obfuscation to Combat Symbolic Execution

Trigger-based code (malicious in many cases, but not necessarily) only executes when specific inputs are received. Symbolic execution has been one of the most powerful techniques in discovering such malicious code and analyzing the trigger condition. We propose a novel automatic malware obfuscation technique to make analysis based on symbolic execution difficult. Unlike previously proposed techniques, the obfuscated code from our tool does not use any cryptographic operations and makes use of only linear operations which symbolic execution is believed to be good in analyzing. The obfuscated code incorporates unsolved conjectures and adds a simple loop to the original code, making it less than one hundred bytes longer and hard to be differentiated from normal programs. Evaluation shows that applying symbolic execution to the obfuscated code is inefficient in finding the trigger condition. We discuss strengths and weaknesses of the proposed technique.

[1]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[2]  Jeffrey C. Lagarias,et al.  The 3x + 1 Problem and its Generalizations , 1985 .

[3]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[4]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[5]  Dawson R. Engler,et al.  RWset: Attacking Path Explosion in Constraint-Based Test Generation , 2008, TACAS.

[6]  Jianying Zhou,et al.  Information and Communications Security , 2013, Lecture Notes in Computer Science.

[7]  John Morris,et al.  Using symbolic execution to guide test generation , 2005, Softw. Test. Verification Reliab..

[8]  Rupak Majumdar,et al.  Testing for buffer overflows with length abstraction , 2008, ISSTA '08.

[9]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[10]  Stephen McCamant,et al.  Extracting Models of Security-Sensitive Operations using String-Enhanced White-Box Exploration on Binaries , 2009 .

[11]  Dawson R. Engler,et al.  Execution Generated Test Cases: How to Make Systems Code Crash Itself , 2005, SPIN.

[12]  Saumya K. Debray,et al.  Obfuscation of executable code to improve resistance to static disassembly , 2003, CCS '03.

[13]  Jonathon T. Giffin,et al.  Impeding Malware Analysis Using Conditional Code Obfuscation , 2008, NDSS.

[14]  David A. Wagner,et al.  Dynamic Test Generation to Find Integer Bugs in x86 Binary Linux Programs , 2009, USENIX Security Symposium.

[15]  Tomasz Zastawniak,et al.  Problem Books in Mathematics , 2004 .

[16]  Manuel Costa,et al.  Bouncer: securing software by blocking bad input , 2008, WRAITS '08.

[17]  Jong Kim,et al.  binOb+: a framework for potent and stealthy binary obfuscation , 2010, ASIACCS '10.

[18]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[19]  David Brumley,et al.  Replayer: automatic protocol replay by binary analysis , 2006, CCS '06.

[20]  Heng Yin,et al.  Panorama: capturing system-wide information flow for malware detection and analysis , 2007, CCS '07.

[21]  R. Guy.  Unsolved Problems in Number Theory , 1981 .

[22]  Richard E. Crandall,et al.  On the $‘‘3x+1”$ problem , 1978 .

[23]  Hao Wang,et al.  Creating Vulnerability Signatures Using Weakest Preconditions , 2007, 20th IEEE Computer Security Foundations Symposium (CSF'07).

[24]  Christian S. Collberg,et al.  A Taxonomy of Obfuscating Transformations , 1997 .

[25]  Stephen McCamant,et al.  Loop-extended symbolic execution on binary programs , 2009, ISSTA.

[26]  Debin Gao,et al.  BinHunt: Automatically Finding Semantic Differences in Binary Programs , 2008, ICICS.

[27]  Christopher Krügel,et al.  Exploring Multiple Execution Paths for Malware Analysis , 2007, 2007 IEEE Symposium on Security and Privacy (SP '07).

[28]  Zhenkai Liang,et al.  Towards Generating High Coverage Vulnerability-Based Signatures with Protocol-Level Constraint-Guided Exploration , 2009, RAID.

[29]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[30]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[31]  Christopher Krügel,et al.  Identifying Dormant Functionality in Malware Programs , 2010, 2010 IEEE Symposium on Security and Privacy.

[32]  Zhenkai Liang,et al.  Automatically Identifying Trigger-based Behavior in Malware , 2008, Botnet Detection.

[33]  Tao Wei,et al.  IntScope: Automatically Detecting Integer Overflow Vulnerability in X86 Binary Using Symbolic Execution , 2009, NDSS.

[34]  Hao Wang,et al.  Towards automatic generation of vulnerability-based signatures , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[35]  David Brumley,et al.  All You Ever Wanted to Know about Dynamic Taint Analysis and Forward Symbolic Execution (but Might Have Been Afraid to Ask) , 2010, 2010 IEEE Symposium on Security and Privacy.

[36]  Christopher Krügel,et al.  Limits of Static Analysis for Malware Detection , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[37]  Gregory R. Andrews,et al.  Binary Obfuscation Using Signals , 2007, USENIX Security Symposium.

[38]  Zhenkai Liang,et al.  BitScope: Automatically Dissecting Malicious Binaries , 2007 .