Unveiling metamorphism by abstract interpretation of code properties

Metamorphic code includes self-modifying semantics-preserving transformations to exploit code diversification. The impact of metamorphism is growing in security and code protection technologies, both for preventing malicious host attacks, e.g., in software diversification for IP and integrity protection, and in malicious software attacks, e.g., in metamorphic malware self-modifying their own code in order to foil detection systems based on signature matching. In this paper we consider the problem of automatically extracting metamorphic signatures from metamorphic code. We introduce a semantics for self-modifying code, later called phase semantics, and prove its correctness by showing that it is an abstract interpretation of the standard trace semantics. Phase semantics precisely models the metamorphic code behavior by providing a set of traces of programs which correspond to the possible evolutions of the metamorphic code during execution. We show that metamorphic signatures can be automatically extracted by abstract interpretation of the phase semantics. In particular, we introduce the notion of regular metamorphism, where the invariants of the phase semantics can be modeled as finite state automata representing the code structure of all possible metamorphic change of a metamorphic code, and we provide a static signature extraction algorithm for metamorphic code where metamorphic signatures are approximated in regular metamorphism.

[1]  Arun Lakhotia,et al.  Static verification of worm and virus behavior in binary executables using model checking , 2003, IEEE Systems, Man and Cybernetics SocietyInformation Assurance Workshop, 2003..

[2]  Jean-Yves Marion,et al.  Server-side dynamic code analysis , 2009, 2009 4th International Conference on Malicious and Unwanted Software (MALWARE).

[3]  Helmut Veith,et al.  Using Verification Technology to Specify and Detect Malware , 2007, EUROCAST.

[4]  Somesh Jha,et al.  Testing malware detectors , 2004, ISSTA '04.

[5]  Eric Filiol,et al.  Metamorphism, Formal Grammars and Undecidable Code Mutation , 2007 .

[6]  Somesh Jha,et al.  A semantics-based approach to malware detection , 2008, TOPL.

[7]  Saumya K. Debray,et al.  Automatic Simplification of Obfuscated JavaScript Code: A Semantics-Based Approach , 2012, 2012 IEEE Sixth International Conference on Software Security and Reliability.

[8]  Philippe Beaucamps,et al.  Advanced Metamorphic Techniques in Computer Viruses , 2007 .

[9]  Thomas W. Reps,et al.  CodeSurfer/x86-A Platform for Analyzing x86 Executables , 2005, CC.

[10]  Thomas W. Reps,et al.  Analyzing Memory Accesses in x86 Executables , 2004, CC.

[11]  Peter Szor,et al.  The Art of Computer Virus Research and Defense , 2005 .

[12]  Magnus O. Myreen.  Verified just-in-time compiler on x86 , 2010, POPL '10.

[13]  H. Read,et al.  Metamorphism , 1940, Nature.

[14]  Nicolas Halbwachs,et al.  Automatic discovery of linear restraints among variables of a program , 1978, POPL.

[15]  Patrick Cousot,et al.  Formal language, grammar and set-constraint-based program analysis by abstract interpretation , 1995, FPCA '95.

[16]  Zhong Shao,et al.  Certified self-modifying code , 2007, PLDI '07.

[17]  Mattia Monga,et al.  Code Normalization for Self-Mutating Malware , 2007, IEEE Security & Privacy.

[18]  Somesh Jha,et al.  Static Analysis of Executables to Detect Malicious Patterns , 2003, USENIX Security Symposium.

[19]  Dawn Xiaodong Song,et al.  Recognizing malicious software behaviors with tree automata inference , 2012, Formal Methods Syst. Des..

[20]  Andrew J. I. Jones.  A Formal Language , 1983 .

[21]  Somesh Jha,et al.  Semantics-aware malware detection , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[22]  Patrick Cousot,et al.  Systematic design of program analysis frameworks , 1979, POPL.

[23]  JhaSomesh,et al.  A semantics-based approach to malware detection , 2007 .

[24]  Stefan Katzenbeisser,et al.  Detecting Malicious Code by Model Checking , 2005, DIMVA.

[25]  Pavel V. Zbitskiy.  Code mutation techniques by means of formal grammars and automatons , 2009, Journal in Computer Virology.

[26]  Roberto Giacobazzi,et al.  Modelling Metamorphism by Abstract Interpretation , 2010, SAS.

[27]  Peter Szor,et al.  HUNTING FOR METAMORPHIC , 2001 .

[28]  Roberto Giacobazzi,et al.  Making abstract interpretations complete , 2000, JACM.

[29]  Jean-Yves Marion,et al.  Behavior Abstraction in Malware Analysis , 2010, RV.

[30]  Arun Lakhotia,et al.  Context-sensitive analysis of obfuscated x86 executables , 2010, PEPM '10.

[31]  Amey Karkare,et al.  Heap reference analysis using access graphs , 2006, ACM Trans. Program. Lang. Syst..

[32]  Mila Dalla Preda.  The Grand Challenge in Metamorphic Analysis , 2012, ICISTM.

[33]  Dawn Xiaodong Song,et al.  Malware Analysis with Tree Automata Inference , 2011, CAV.

[34]  Arto Salomaa,et al.  Formal languages , 1973, Computer science classics.

[35]  Patrick Cousot,et al.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints , 1977, POPL.