VMAttack: Deobfuscating Virtualization-Based Packed Binaries

We present VMAttack, a deobfuscation tool for virtualization-packed binaries based on automated static and dynamic analysis, which offers a simplified view of the disassembly. VMAttack is implemented as a plug-in for IDA Pro and as such, integrates seamlessly with manual reverse engineering. The complexity of the disassembly view is notably reduced by analyzing the inner working principles of the VM layer of protected binaries. Using static analysis, complex bytecode sequences of the VM are mapped to easy-to-read pseudo-code instructions, based on an intermediate representation specifically designed for stack-based virtual machines. Using dynamic analysis, we identify structural components like the interpreter loop and compress instruction sequences by filtering out semantically redundant instructions of the execution trace. The integrated result, which rates both static and dynamic analysis's results, provides the reverse engineer with a deobfuscated disassembly that tolerates weaknesses of a single analysis technique. VMAttack is currently limited to stack-based virtual machines like VMProtect. We evaluated VMAttack using binaries obfuscated with VMProtect and achieved an average execution trace reduction of 89.86% for the dynamic and 96.67% for the combined static and dynamic analysis.

[1]  Christopher Krügel,et al.  Driller: Augmenting Fuzzing Through Selective Symbolic Execution , 2016, NDSS.

[2]  Bart Preneel,et al.  Cryptanalysis of a Perturbated White-Box AES Implementation , 2010, INDOCRYPT.

[3]  Johannes Kinder.  Towards Static Analysis of Virtualization-Obfuscated Binaries , 2012, 2012 19th Working Conference on Reverse Engineering.

[4]  Christian S. Collberg,et al.  Surreptitious Software - Obfuscation, Watermarking, and Tamperproofing for Software Protection , 2009, Addison-Wesley Software Security Series.

[5]  Eldad Eilam,et al.  Reversing: Secrets of Reverse Engineering , 2005 .

[6]  Heng Yin,et al.  Renovo: a hidden code extractor for packed executables , 2007, WORM '07.

[7]  Iain D. Craig,et al.  Virtual machines , 2005 .

[8]  Joshua Jaffe,et al.  SELF-PROTECTING DIGITAL CONTENT , 2003 .

[9]  Yoann Guillot,et al.  Automatic binary deobfuscation , 2009, Journal in Computer Virology.

[10]  Jonathon T. Giffin,et al.  Automatic Reverse Engineering of Malware Emulators , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[11]  Ramarathnam Venkatesan,et al.  Proteus: virtualization for diversified tamper-resistance , 2006, DRM '06.

[12]  Suhaimi Ibrahim,et al.  Camouflage in Malware: from Encryption to Metamorphism , 2012 .

[13]  Saumya Debray,et al.  A Generic Approach to Automatic Deobfuscation of Executable Code , 2015, 2015 IEEE Symposium on Security and Privacy.

[14]  Joseph Robert Horgan,et al.  Dynamic program slicing , 1990, PLDI '90.

[15]  Kevin Coogan.  Deobfuscation of Packed and Virtualization-Obfuscation Protected Binaries , 2011 .

[16]  Ravi Nair,et al.  System Virtual Machines , 2005 .

[17]  Christopher Krügel,et al.  SOK: (State of) The Art of War: Offensive Techniques in Binary Analysis , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[18]  Shuhong Wang,et al.  Multi-stage Binary Code Obfuscation Using Improved Virtual Machine , 2011, ISC.

[19]  Rolf Rolles,et al.  Unpacking Virtualization Obfuscators , 2009, WOOT.

[20]  Kevin Coogan,et al.  Deobfuscation of virtualization-obfuscated software: a semantics-based approach , 2011, CCS '11.

[21]  James E. Smith,et al.  Virtual machines - versatile platforms for systems and processes , 2005 .