On Emulation-Based Network Intrusion Detection Systems

Emulation-based network intrusion detection systems have been devised to detect the presence of shellcode in network traffic by trying to execute (portions of) the network packet payloads in an in- strumented environment and checking the execution traces for signs of shellcode activity. Emulation-based network intrusion detection systems are regarded as a significant step forward with regards to traditional signature-based systems, as they allow detecting polymorphic (i.e., en- crypted) shellcode. In this paper we investigate and test the actual effec- tiveness of emulation-based detection and show that the detection can be circumvented by employing a wide range of evasion techniques, ex- ploiting weakness that are present at all three levels in the detection process. We draw the conclusion that current emulation-based systems have limitations that allow attackers to craft generic shellcode encoders able to circumvent their detection mechanisms.

[1]  Peng Ning,et al.  Analyzing network traffic to detect self-decrypting exploit code , 2007, ASIACCS '07.

[2]  Christian S. Collberg,et al.  Protecting Against Unexpected System Calls , 2005, USENIX Security Symposium.

[3]  Peter Ferrie.  Attacks on More Virtual Machine Emulators , 2007 .

[4]  Niels Provos,et al.  SHELLOS: Enabling Fast Detection and Forensic Analysis of Code Injection Attacks , 2011, USENIX Security Symposium.

[5]  Christopher Krügel,et al.  Detecting System Emulators , 2007, ISC.

[6]  John Aycock,et al.  Anti-disassembly using Cryptographic Hash Functions , 2006, Journal in Computer Virology.

[7]  Evangelos P. Markatos,et al.  Network-level polymorphic shellcode detection using emulation , 2006, Journal in Computer Virology.

[8]  Ronald L. Rivest,et al.  Time-lock Puzzles and Timed-release Crypto , 1996 .

[9]  Aloysius K. Mok,et al.  Swarm Attacks against Network-Level Emulation/Analysis , 2008, RAID.

[10]  Christopher Krügel,et al.  Defending Browsers against Drive-by Downloads: Mitigating Heap-Spraying Code Injection Attacks , 2009, DIMVA.

[11]  Leyla Bilge,et al.  G-Free: defeating return-oriented programming through gadget-less binaries , 2010, ACSAC '10.

[12]  Piotr Bania.  Evading network-level emulation , 2009, ArXiv.

[13]  B. Harrison.  Las Vegas, Nevada , 2002 .

[14]  Evangelos P. Markatos,et al.  Comprehensive shellcode detection using runtime heuristics , 2010, ACSAC '10.

[15]  Dong Xuan,et al.  Malicious Shellcode Detection with Virtual Memory Snapshots , 2010, 2010 Proceedings IEEE INFOCOM.

[16]  Andrew Honig,et al.  Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software , 2012 .

[17]  Evangelos P. Markatos,et al.  Emulation-Based Detection of Non-self-contained Polymorphic Shellcode , 2007, RAID.

[18]  Angelos D. Keromytis,et al.  ROP payload detection using speculative code execution , 2011, 2011 6th International Conference on Malicious and Unwanted Software.

[19]  Makoto Shimamura,et al.  Yataglass: Network-Level Code Emulation for Analyzing Memory-Scanning Attacks , 2009, DIMVA.

[20]  Gabriel Negreira Barbosa,et al.  Scientific but Not Academical Overview of Malware Anti-Debugging , Anti-Disassembly and Anti-VM Technologies , 2012 .

[21]  Herbert Bos,et al.  Argos: an emulator for fingerprinting zero-day attacks for advertised honeypots with automatic signature generation , 2006, EuroSys.

[22]  Pavel Laskov,et al.  Detection of Intrusions and Malware, and Vulnerability Assessment: 19th International Conference, DIMVA 2022, Cagliari, Italy, June 29 –July 1, 2022, Proceedings , 2022, International Conference on Detection of intrusions and malware, and vulnerability assessment.

[23]  Lucas Davi,et al.  ROPdefender: a detection tool to defend against return-oriented programming attacks , 2011, ASIACCS '11.

[24]  Bing Mao,et al.  DROP: Detecting Return-Oriented Programming Malicious Code , 2009, ICISS.