A Native APIs Protection Mechanism in the Kernel Mode against Malicious Code

As new vulnerabilities on Windows systems are reported endlessly, it is more practical to stop polymorphic malicious code from exploiting these vulnerabilities by building an behavior-based monitor, rather than adopting a signature-based detection system or fixing these vulnerabilities. Many behavior-based monitors have been proposed for Windows systems to serve this purpose. Some of them hook high-level system APIs to detect the suspicious behaviors of code. However, they cannot detect malicious code that directly invokes Native APIs. In this paper, we present a novel security scheme that hooks Native APIs in the kernel mode. This method effectively prevents malicious code calling Native APIs directly. It introduces an average eight percent computation overhead into the system. Analyses and a series of experiments are given in the paper to support our claims.

[1]  Christian S. Collberg,et al.  Protecting Against Unexpected System Calls , 2005, USENIX Security Symposium.

[2]  Yanfang Ye,et al.  IMDS: intelligent malware detection system , 2007, KDD '07.

[3]  Tzi-cker Chiueh,et al.  RAD: a compile-time solution to buffer overflow attacks , 2001, Proceedings 21st International Conference on Distributed Computing Systems.

[4]  Crispan Cowan,et al.  StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks , 1998, USENIX Security Symposium.

[5]  Hovav Shacham,et al.  On the effectiveness of address-space randomization , 2004, CCS '04.

[6]  Crispin Cowan,et al.  FormatGuard: Automatic Protection From printf Format String Vulnerabilities , 2001, USENIX Security Symposium.

[7]  Hung-Min Sun,et al.  API Monitoring System for Defeating Worms and Exploits in MS-Windows System , 2006, ACISP.

[8]  Karl N. Levitt,et al.  A framework for diversifying windows native APIs to tolerate code injection attacks , 2007, ASIACCS '07.

[9]  Luigi V. Mancini,et al.  A Host Intrusion Prevention System for Windows Operating Systems , 2004, ESORICS.

[10]  Christopher Krügel,et al.  Accurate Buffer Overflow Detection via Abstract Payload Execution , 2002, RAID.

[11]  John Johansen,et al.  PointGuard™: Protecting Pointers from Buffer Overflow Vulnerabilities , 2003, USENIX Security Symposium.

[12]  Matti A. Hiltunen,et al.  Authenticated system calls , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[13]  Richard D. Schlichting,et al.  Signed system calls and hidden fingerprints , 2004 .

[14]  David Evans,et al.  Statically Detecting Likely Buffer Overflow Vulnerabilities , 2001, USENIX Security Symposium.

[15]  Udo Payer,et al.  Hybrid Engine for Polymorphic Shellcode Detection , 2005, DIMVA.

[16]  Galen C. Hunt,et al.  Detours: binary interception of Win32 functions , 1999 .

[17]  Evangelos P. Markatos,et al.  STRIDE: Polymorphic Sled Detection through Instruction Sequence Analysis , 2005, SEC.

[18]  David A. Wagner,et al.  Mimicry attacks on host-based intrusion detection systems , 2002, CCS '02.

[19]  Jesse C. Rabek,et al.  Detection of injected, dynamically generated, and obfuscated malicious code , 2003, WORM '03.

[20]  Cheng Zhang,et al.  Native API based Windows anomaly intrusion detection method using SVM , 2006, IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC'06).