Practical Protection of Kernel Integrity for Commodity OS from Untrusted Extensions

Kernel extensions are widely used by attackers to compromise the operating system kernel. With the presence of various untrusted extensions, it remains a challenging problem to comprehensively preserve the integrity of OS kernels in a practical and generic way. In this paper, we present HUKO, a hypervisor-based integrity protection system designed to protect commodity OS kernels from untrusted extensions. In HUKO system, untrusted kernel extensions can safely run to provide desired functionalities. The behaviors of untrusted extensions, however, are confined by mandatory access control policies, which significantly limit the attacker’s ability to compromise the integrity of the kernel. To guarantee multi-aspect protection and enforcement, HUKO leverages hardware assisted paging to transparently isolate untrusted extensions from the OS kernel. Moreover, HUKO overcomes the challenge of mediation overhead by introducing a novel design named subject-aware protection state transition to eliminate unnecessary privilege transitions caused by mediating allowed accesses. Our approach is practical because it requires little change for either OS kernel or extensions, and it can inherently support multiple commodity operating systems and legacy extensions. We have implemented a prototype of HUKO based on the open source Xen hypervisor. The evaluation results show that HUKO can comprehensively protect the integrity for both Linux and Windows kernel from various kinds of malicious extensions with an acceptable performance cost.

[1]  Calton Pu,et al.  Soft-Timer Driven Transient Kernel Control Flow Attacks and Defense , 2008, 2008 Annual Computer Security Applications Conference (ACSAC).

[2]  James R. Larus,et al.  Language support for fast and reliable message-based communication in singularity OS , 2006, EuroSys.

[3]  George C. Necula,et al.  SafeDrive: safe and recoverable extensions using language-based techniques , 2006, OSDI '06.

[4]  Xuxian Jiang,et al.  Multi-aspect profiling of kernel rootkit behavior , 2009, EuroSys '09.

[5]  Hong Chen,et al.  Usable Mandatory Integrity Protection for Operating Systems , 2007, 2007 IEEE Symposium on Security and Privacy (SP '07).

[6]  Krste Asanovic,et al.  Mondrian memory protection , 2002, ASPLOS X.

[7]  Christoforos E. Kozyrakis,et al.  Hardware Enforcement of Application Security Policies Using Tagged Memory , 2008, OSDI.

[8]  Wenke Lee,et al.  Secure in-VM monitoring using hardware virtualization , 2009, CCS.

[9]  Adrian Perrig,et al.  SecVisor: a tiny hypervisor to provide lifetime kernel code integrity for commodity OSes , 2007, SOSP.

[10]  Zhenkai Liang,et al.  HookFinder: Identifying and Understanding Malware Hooking Behaviors , 2008, NDSS.

[11]  Krste Asanovic,et al.  Mondrix: memory isolation for linux using mondriaan memory protection , 2005, SOSP '05.

[12]  Herbert Bos,et al.  Fault isolation for device drivers , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[13]  Michael Norrish,et al.  seL4: formal verification of an OS kernel , 2009, SOSP '09.

[14]  Xuxian Jiang,et al.  Countering kernel rootkits with lightweight hook protection , 2009, CCS.

[15]  Timothy Fraser,et al.  LOMAC: Low Water-Mark integrity protection for COTS environments , 2000, Proceeding 2000 IEEE Symposium on Security and Privacy. S&P 2000.

[16]  Gil Neiger,et al.  Intel ® Virtualization Technology for Directed I/O , 2006 .

[17]  Abhinav Srivastava,et al.  Robust signatures for kernel data structures , 2009, CCS.

[18]  Xuxian Jiang,et al.  Guest-Transparent Prevention of Kernel Rootkits with VMM-Based Memory Shadowing , 2008, RAID.

[19]  David Lie,et al.  Hypervisor Support for Identifying Covertly Executing Binaries , 2008, USENIX Security Symposium.

[20]  Martín Abadi,et al.  XFI: software guards for system address spaces , 2006, OSDI '06.

[21]  Abhinav Srivastava,et al.  Kernel Data Integrity Protection via Memory Access Control , 2009 .

[22]  Felix C. Freiling,et al.  Return-Oriented Rootkits: Bypassing Kernel Code Integrity Protection Mechanisms , 2009, USENIX Security Symposium.

[23]  Michael W. Hicks,et al.  Automated detection of persistent kernel control-flow attacks , 2007, CCS '07.

[24]  Neha Narula,et al.  Native Client: A Sandbox for Portable, Untrusted x86 Native Code , 2009, IEEE Symposium on Security and Privacy.

[25]  Ruby B. Lee,et al.  Scalable architectural support for trusted software , 2010, HPCA - 16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture.

[26]  Arati Baliga,et al.  Automatic Inference and Enforcement of Kernel Data Structure Invariants , 2008, 2008 Annual Computer Security Applications Conference (ACSAC).

[27]  Brian N. Bershad,et al.  Improving the reliability of commodity operating systems , 2005, TOCS.

[28]  Zhi Wang,et al.  Countering Persistent Kernel Rootkits through Systematic Hook Discovery , 2008, RAID.

[29]  Heng Yin,et al.  Panorama: capturing system-wide information flow for malware detection and analysis , 2007, CCS '07.

[30]  Hovav Shacham,et al.  The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86) , 2007, CCS '07.

[31]  J. Shapiro,et al.  EROS: a fast capability system , 2000, OPSR.

[32]  Xiaoxin Chen,et al.  Overshadow: a virtualization-based approach to retrofitting protection in commodity operating systems , 2008, ASPLOS.

[33]  Wenke Lee,et al.  K-Tracer: A System for Extracting Kernel Malware Behavior , 2009, NDSS.

[34]  Xuxian Jiang,et al.  Mapping kernel objects to enable systematic integrity checking , 2009, CCS.