Abstraction-Based Malware Analysis Using Rewriting and Model Checking

We propose a formal approach for the detection of high-level malware behaviors. Our technique uses a rewriting-based abstraction mechanism, producing abstracted forms of program traces, independent of the program implementation. It then allows us to handle similar behaviors in a generic way and thus to be robust with respect to variants. These behaviors, defined as combinations of patterns given in a signature, are detected by model-checking on the high-level representation of the program. We work on unbounded sets of traces, which makes our technique useful not only for dynamic analysis, considering one trace at a time, but also for static analysis, considering a set of traces inferred from a control flow graph. Abstracting traces with rewriting systems on first order terms with variables allows us in particular to model dataflow and to detect information leak.

[1]  Helmut Veith,et al.  An Abstract Interpretation-Based Framework for Control Flow Reconstruction from Binaries , 2008, VMCAI.

[2]  Eric Filiol,et al.  Malware Behavioral Detection by Attribute-Automata Using Abstraction from Platform and Language , 2009, RAID.

[3]  Baudouin Le Charlier,et al.  Dynamic Detection and Classification of Computer Viruses Using General Behaviour Patterns , 1995 .

[4]  Hubert Comon,et al.  Tree automata techniques and applications , 1997 .

[5]  Christopher Krügel,et al.  Scalable, Behavior-Based Malware Clustering , 2009, NDSS.

[6]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[7]  Peter J. Clarke,et al.  Characterization of virus replication , 2007, Journal in Computer Virology.

[8]  Christophe Devine.  A study of anti-virus ’ response to unknown threats , 2009 .

[9]  Somesh Jha,et al.  A Layered Architecture for Detecting Malicious Behaviors , 2008, RAID.

[10]  Stephan Merz,et al.  Temporal Logic and State Systems , 2008, Texts in Theoretical Computer Science. An EATCS Series.

[11]  Shin Nakajima,et al.  The SPIN Model Checker : Primer and Reference Manual , 2004 .

[12]  Fred Cohen,et al.  Computer viruses—theory and experiments , 1990 .

[13]  Somesh Jha,et al.  Semantics-aware malware detection , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[14]  Christopher Krügel,et al.  Behavior-based Spyware Detection , 2006, USENIX Security Symposium.

[15]  Arun Lakhotia,et al.  Static verification of worm and virus behavior in binary executables using model checking , 2003, IEEE Systems, Man and Cybernetics SocietyInformation Assurance Workshop, 2003..

[16]  Stephan Merz,et al.  Temporal Logic and State Systems (Texts in Theoretical Computer Science. An EATCS Series) , 2010 .

[17]  Grigore Rosu,et al.  Mop: an efficient and generic runtime verification framework , 2007, OOPSLA.

[18]  Jean-Yves Marion,et al.  Behavior Analysis of Malware by Rewriting-based Abstraction - Extended Version , 2011 .

[19]  Somesh Jha,et al.  A semantics-based approach to malware detection , 2008, TOPL.

[20]  Stefan Katzenbeisser,et al.  Detecting Malicious Code by Model Checking , 2005, DIMVA.

[21]  Neha Narula,et al.  Native Client: A Sandbox for Portable, Untrusted x86 Native Code , 2009, IEEE Symposium on Security and Privacy.

[22]  Kousha Etessami,et al.  Analysis of Recursive Game Graphs Using Data Flow Equations , 2004, VMCAI.

[23]  Zhenkai Liang,et al.  BitScope: Automatically Dissecting Malicious Binaries , 2007 .

[24]  R. Sekar,et al.  A fast automaton-based method for detecting anomalous program behaviors , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[25]  Jules Desharnais,et al.  Static Detection of Malicious Code in Executable Programs , 2000 .

[26]  Sophie Tison,et al.  Regular Tree Languages and Rewrite Systems , 1995, Fundam. Informaticae.

[27]  Jean-Yves Marion,et al.  Behavior Abstraction in Malware Analysis , 2010, RV.

[28]  Radu Mateescu,et al.  CADP 2010: A Toolbox for the Construction and Analysis of Distributed Processes , 2011, TACAS.