VMGuard: A VMI-Based Security Architecture for Intrusion Detection in Cloud Environment

Cloud security is of paramount importance in the new era of computing. Advanced malware can hide their behavior on detection of the presence of a security tool at a tenant virtual machine (TVM). Hence, TVM-layer security solutions are not reliable. In this paper, we propose a Virtual Machine Introspection (VMI) based security architecture design for fine granular monitoring of the virtual machines to detect known attacks and their variants. We have developed techniques for monitoring the TVMs at the process level and system call level to detect attacks such as those based on malicious hidden processes, attacks that disable security tools in the virtual machines and attacks that alter the behavior of legitimate applications to access sensitive data. Our architecture, VMGuard, utilizes the introspection feature at the VMM-layer to analyze system call traces of programs running on TVM. VMGuard applies the software breakpoint injection technique which is OS agnostic and can be used to trap the execution of programs. Motivated by text mining approaches, VMGuard provides ‘Bag of n-grams (BonG)’ approach integrated with Term Frequency-Inverse Document Frequency (TF-IDF) method, to extract and select features of normal and attack traces. It then applies the Random Forest classifier to produce a generic behavior for different categories of intrusions of the monitored TVM. We have implemented a prototype and conducted a detailed analysis using University of New Mexico (UNM) datasets and a Windows malware dataset obtained from the University of California. The results obtained are promising and demonstrate the applicability of the VMGuard. We compare VMGuard with existing techniques and discuss its advantages.

[1]  Peter Ferrie.  Attacks on More Virtual Machine Emulators , 2007 .

[2]  Marino Miculan,et al.  Unobservable intrusion detection based on call traces in paravirtualized systems , 2011, Proceedings of the International Conference on Security and Cryptography.

[3]  V. Rao Vemuri,et al.  Using Text Categorization Techniques for Intrusion Detection , 2002, USENIX Security Symposium.

[4]  Andrea C. Arpaci-Dusseau,et al.  VMM-based hidden process detection and identification using Lycosid , 2008, VEE '08.

[5]  Kevin Leach,et al.  LO-PHI: Low-Observable Physical Host Instrumentation for Malware Analysis , 2016, NDSS.

[6]  Bryan D. Payne,et al.  Simplifying virtual machine introspection using LibVMI. , 2012 .

[7]  Aggelos Kiayias,et al.  Scalability, fidelity and stealth in the DRAKVUF dynamic malware analysis system , 2014, ACSAC.

[8]  Hannes Federrath,et al.  Website fingerprinting: attacking popular privacy enhancing technologies with the multinomial naïve-bayes classifier , 2009, CCSW '09.

[9]  Xuxian Jiang,et al.  Stealthy malware detection through vmm-based "out-of-the-box" semantic view reconstruction , 2007, CCS '07.

[10]  Dae-Ki Kang,et al.  Learning classifiers for misuse and anomaly detection using a bag of system calls representation , 2005, Proceedings from the Sixth Annual IEEE SMC Information Assurance Workshop.

[11]  Vijay Varadharajan,et al.  VAED: VMI‐assisted evasion detection approach for infrastructure as a service cloud , 2017, Concurr. Comput. Pract. Exp..

[12]  Claudia Eckert,et al.  Nitro: Hardware-Based System Call Tracing for Virtual Machines , 2011, IWSEC.

[13]  Max Mühlhäuser,et al.  A framework for evaluating trust of service providers in cloud marketplaces , 2013, SAC '13.

[14]  Stephen D. Wolthusen,et al.  Anomaly Detection for Ephemeral Cloud IaaS Virtual Machines , 2013, NSS.

[15]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[16]  Kam-Fai Wong,et al.  Interpreting TF-IDF term weights as making relevance decisions , 2008, TOIS.

[17]  Xiaohong Guan,et al.  An SVM-based machine learning method for accurate internet traffic classification , 2010, Inf. Syst. Frontiers.

[18]  Padam Kumar,et al.  System cum Program-Wide Lightweight Malicious Program Execution Detection Scheme for Cloud , 2014, Inf. Secur. J. A Glob. Perspect..

[19]  Wenke Lee,et al.  K-Tracer: A System for Extracting Kernel Malware Behavior , 2009, NDSS.

[20]  Tal Garfinkel,et al.  A Virtual Machine Introspection Based Architecture for Intrusion Detection , 2003, NDSS.

[21]  Christopher Krügel,et al.  BareCloud: Bare-metal Analysis-based Evasive Malware Detection , 2014, USENIX Security Symposium.

[22]  Lei Wang,et al.  AdaBoost with SVM-based component classifiers , 2008, Eng. Appl. Artif. Intell..

[23]  Vijay Varadharajan,et al.  On the Design and Implementation of an Integrated Security Architecture for Cloud with Improved Resilience , 2017, IEEE Transactions on Cloud Computing.

[24]  Vijay Varadharajan,et al.  Intrusion detection techniques in cloud environment: A survey , 2017, J. Netw. Comput. Appl..

[25]  Zach Hanif,et al.  Internet-Scale File Analysis , 2015 .

[26]  Andrea C. Arpaci-Dusseau,et al.  Antfarm: Tracking Processes in a Virtual Machine Environment , 2006, USENIX Annual Technical Conference, General Track.

[27]  Barak A. Pearlmutter,et al.  Detecting intrusions using system calls: alternative data models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[28]  Stephanie Forrest,et al.  Intrusion Detection Using Sequences of System Calls , 1998, J. Comput. Secur..

[29]  David Lie,et al.  Hypervisor Support for Identifying Covertly Executing Binaries , 2008, USENIX Security Symposium.

[30]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[31]  Wenke Lee,et al.  Secure and Flexible Monitoring of Virtual Machines , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[32]  Padam Kumar,et al.  An Immediate System Call Sequence Based Approach for Detecting Malicious Program Executions in Cloud Environment , 2015, Wirel. Pers. Commun..

[33]  Stephen D. Wolthusen,et al.  Detecting anomalies in IaaS environments through virtual machine host system call analysis , 2012, 2012 International Conference for Internet Technology and Secured Transactions.

[34]  Li Dong,et al.  Feature representation and selection in malicious code detection methods based on static system calls , 2011, Comput. Secur..

[35]  Radu Sion,et al.  SoK: Introspections on Trust and the Semantic Gap , 2014, 2014 IEEE Symposium on Security and Privacy.

[36]  David Hutchison,et al.  Malware Detection in Cloud Computing Infrastructures , 2016, IEEE Transactions on Dependable and Secure Computing.

[37]  Peng Liu,et al.  System Call Redirection: A Practical Approach to Meeting Real-World Virtual Machine Introspection Needs , 2014, 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks.

[38]  Wenke Lee,et al.  Lares: An Architecture for Secure Active Monitoring Using Virtualization , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).