Reversing Compiled Executables for Malware Analysis via Visualization

Reverse engineering-compiled executables is a task with a steep learning curve. It is complicated by the task of translating assembly into a series of abstractions that represent the overall flow of a program. Most of the steps involve finding interesting areas of an executable and determining their general functionality. This article presents a method using dynamic analysis of program execution to visually represent the general flow of a program. We use the Ether hypervisor framework to covertly monitor a program. The data are processed and presented for the reverse engineer. The VERA (Visualization of Executables for Reversing and Analysis) system specifically accelerates the location of the original entry point and understanding of overall executable functionality. Using this method, the amount of time needed to extract key features of an executable is greatly reduced, improving productivity. Two malware samples are used to demonstrate the advantages of using the VERA system to reverse engineer malware. Further, these examples exemplify a reversing process enhanced through effective use of dynamic analysis tools. Preliminary user study indicates that the tool is useful for both new and experienced users.

[1]  U. Bayer,et al.  TTAnalyze: A Tool for Analyzing Malware , 2006 .

[2]  Peter Ferrie.  Attacks on Virtual Machine Emulators , 2007 .

[3]  Henry L. Owen,et al.  Visual Analysis of Program Flow Data with Data Propagation , 2008, VizSEC.

[4]  Helen J. Wang,et al.  Finding diversity in remote code injection exploits , 2006, IMC '06.

[5]  Jürgen Döllner,et al.  Visual exploration of function call graphs for feature location in complex software systems , 2006, SoftVis '06.

[6]  Tzi-cker Chiueh,et al.  A Study of the Packer Problem and Its Solutions , 2008, RAID.

[7]  Amit Vasudevan,et al.  SPiKE: engineering malware analysis tools using unobtrusive binary-instrumentation , 2006, ACSC.

[8]  Heng Yin,et al.  Renovo: a hidden code extractor for packed executables , 2007, WORM '07.

[9]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[10]  Lorie M. Liebrock,et al.  Visualizing compiled executables for malware analysis , 2009, 2009 6th International Workshop on Visualization for Cyber Security.

[11]  Cristina Cifuentes,et al.  Computer security analysis through decompilation and high-level debugging , 2001, Proceedings Eighth Working Conference on Reverse Engineering.

[12]  Wenke Lee,et al.  Ether: malware analysis via hardware virtualization extensions , 2008, CCS.

[13]  Wenke Lee,et al.  PolyUnpack: Automating the Hidden-Code Extraction of Unpack-Executing Malware , 2006, 2006 22nd Annual Computer Security Applications Conference (ACSAC'06).

[14]  Lucian Voinea,et al.  An interactive reverse engineering environment for large-scale C++ code , 2008, SoftVis '08.