Malware and Formal Methods: Rigorous Approaches for detecting Malicious Behaviour

The crucial aim of software security is malware detection. A malware is a program with malicious intents. The predominate anti-malware solutions are signature-based. These detectors compute the signature starting from the syntactic characteristics of the malicious code. Unfortunately, the signature-based techniques are ineffective against the code obfuscations, i.e., trivial transformations that alter the syntax of the code preserving the normal behaviour of the program. To address this limitation, formal methods are used in software security. Formal methods are rigorous techniques used to verify the behaviour of a system. This paper aims to make an overview on behavioural based techniques developed to detect malware programs. The illustrated approaches are based on different formal techniques.

[1]  Arnaldo Moura,et al.  Automated Malware Invariant Generation , 2009 .

[2]  E. Allen Emerson,et al.  Temporal and Modal Logic , 1991, Handbook of Theoretical Computer Science, Volume B: Formal Models and Sematics.

[3]  J. Van Leeuwen,et al.  Handbook of theoretical computer science - Part A: Algorithms and complexity; Part B: Formal models and semantics , 1990 .

[4]  Helmut Veith,et al.  Jakstab: A Static Analysis Platform for Binaries , 2008, CAV.

[5]  Natarajan Shankar,et al.  PVS: A Prototype Verification System , 1992, CADE.

[6]  Somesh Jha,et al.  Semantics-aware malware detection , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[7]  Stephan Merz,et al.  Model Checking , 2000 .

[8]  Gerardo Canfora,et al.  Composition-Malware: Building Android Malware at Run Time , 2015, 2015 10th International Conference on Availability, Reliability and Security.

[9]  Robin Milner,et al.  Handbook of Theoretical Computer Science (Vol. B) , 1990 .

[10]  Colin Stirling,et al.  An Introduction to Modal and Temporal Logics for CCS , 1991, Concurrency: Theory, Language, And Architecture.

[11]  Stefan Katzenbeisser,et al.  Proactive Detection of Computer Worms Using Model Checking , 2010, IEEE Transactions on Dependable and Secure Computing.

[12]  Tayssir Touili,et al.  Efficient Malware Detection Using Model-Checking , 2012, FM.

[13]  Patrick Cousot,et al.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints , 1977, POPL.

[14]  Alessandro Coglio,et al.  Android Platform Modeling and Android App Verification in the ACL2 Theorem Prover , 2015, VSTTE.

[15]  Somesh Jha,et al.  A semantics-based approach to malware detection , 2007, POPL '07.

[16]  Olivier Ly,et al.  The BINCOA Framework for Binary Code Analysis , 2011, CAV.

[17]  Robin Milner,et al.  Communication and concurrency , 1989, PHI Series in computer science.

[18]  Tayssir Touili,et al.  Pushdown model checking for malware detection , 2013, International Journal on Software Tools for Technology Transfer.

[19]  J. Strother Moore,et al.  An Industrial Strength Theorem Prover for a Logic Based on Common Lisp , 1997, IEEE Trans. Software Eng..

[20]  Serge Chaumette,et al.  Automated extraction of polymorphic virus signatures using abstract interpretation , 2011, 2011 5th International Conference on Network and System Security.