NumChecker: Detecting kernel control-flow modifying rootkits by using Hardware Performance Counters

This paper presents NumChecker, a new Virtual Machine Monitor (VMM) based framework to detect control-flow modifying kernel rootkits in a guest Virtual Machine (VM). NumChecker detects malicious modifications to a system call in the guest VM by checking the number of certain hardware events that occur during the system call's execution. To automatically count these events, NumChecker leverages the Hardware Performance Counters (HPCs), which exist in most modern processors. By using HPCs, the checking cost is significantly reduced and the tamper-resistance is enhanced. We implement a prototype of NumChecker on Linux with the Kernel-based Virtual Machine (KVM). Our evaluation demonstrates its practicality and effectiveness.

[1]  Claudia Eckert,et al.  Nitro: Hardware-Based System Call Tracing for Virtual Machines , 2011, IWSEC.

[2]  Xuxian Jiang,et al.  Countering kernel rootkits with lightweight hook protection , 2009, CCS.

[3]  David Kaeli,et al.  Virtual machine monitor-based lightweight intrusion detection , 2011, OPSR.

[4]  刘锋,et al.  Kernel-based virtual machine事件跟踪机制的设计与实现 , 2008 .

[5]  Xuxian Jiang,et al.  Guest-Transparent Prevention of Kernel Rootkits with VMM-Based Memory Shadowing , 2008, RAID.

[6]  Zhi Wang,et al.  DKSM: Subverting Virtual Machine Introspection for Fun and Profit , 2010, 2010 29th IEEE Symposium on Reliable Distributed Systems.

[7]  Tal Garfinkel,et al.  A Virtual Machine Introspection Based Architecture for Intrusion Detection , 2003, NDSS.

[8]  Wenke Lee,et al.  Ether: malware analysis via hardware virtualization extensions , 2008, CCS.

[9]  Emmett Witchel,et al.  Ensuring operating system kernel integrity with OSck , 2011, ASPLOS XVI.

[10]  Wenke Lee,et al.  Lares: An Architecture for Secure Active Monitoring Using Virtualization , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[11]  Ramesh Karri,et al.  Are hardware performance counters a cost effective way for integrity checking of programs , 2011, STC '11.

[12]  Wenke Lee,et al.  Secure and Flexible Monitoring of Virtual Machines , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[13]  Michael W. Hicks,et al.  Automated detection of persistent kernel control-flow attacks , 2007, CCS '07.

[14]  Xuxian Jiang,et al.  Stealthy malware detection through vmm-based "out-of-the-box" semantic view reconstruction , 2007, CCS '07.

[15]  Abhinav Srivastava,et al.  Tamper-Resistant, Application-Aware Blocking of Malicious Network Connections , 2008, RAID.