Behavior based authentication mechanism to prevent malicious code attacks in windows

Most modern kernel of the operating system fails to ensure the authenticity of a suspicious process while servicing its system call. As a result, preventing kernel level malicious code attacks that target system table hooking becomes a challenging and serious security issue. The traditional process authentication techniques such as the process name, process identifier and execution path exercised by the kernel are not reliable. Therefore, in this paper, we proposed a kernel level authentication prototype to verify the originality of each suspicious process during runtime. The verification and authentication tasks are performed well in advance before each suspicious process getting the kernel service. We designed, implemented, and assessed the prototype in Windows. The evaluation results confirm that the prototype successfully blocked all malicious code attacks that target invoking system services directly in the kernel mode with minimal overhead.

[1]  Daniel C. DuVarney,et al.  Model-carrying code , 2003, SOSP 2003.

[2]  Tal Garfinkel,et al.  Ostia: A Delegating Architecture for Secure System Call Interposition , 2004, NDSS.

[3]  Trent Jaeger,et al.  Design and Implementation of a TCG-based Integrity Measurement Architecture , 2004, USENIX Security Symposium.

[4]  Hong Chen,et al.  Analyzing and Comparing the Protection Quality of Security Enhanced Operating Systems , 2009, NDSS.

[5]  Trent Jaeger,et al.  Measuring integrity on mobile phone systems , 2008, SACMAT '08.

[6]  Massimo Bernaschi,et al.  Operating system enhancements to prevent the misuse of system calls , 2000, CCS.

[7]  Trent Jaeger,et al.  PRIMA: policy-reduced integrity measurement architecture , 2006, SACMAT '06.

[8]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[9]  Peng Ning,et al.  Remote attestation to dynamic system properties: Towards providing complete system integrity evidence , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[10]  Daniel C. DuVarney,et al.  Model-carrying code: a practical approach for safe execution of untrusted applications , 2003, SOSP '03.

[11]  Tal Garfinkel,et al.  Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools , 2003, NDSS.

[12]  Matti A. Hiltunen,et al.  System Call Monitoring Using Authenticated System Calls , 2006, IEEE Transactions on Dependable and Secure Computing.

[13]  Stephen Smalley,et al.  Integrating Flexible Support for Security Policies into the Linux Operating System , 2001, USENIX Annual Technical Conference, FREENIX Track.

[14]  R. Sekar,et al.  User-Level Infrastructure for System Call Interposition: A Platform for Intrusion Detection and Confinement , 2000, NDSS.