Trusted system-calls analysis methodology aimed at detection of compromised virtual machines using sequential mining

Abstract Most organizations today employ cloud-computing environments and virtualization technology; Due to their prevalence and importance in providing services to the entire organization, virtual-servers are constantly targeted by cyber-attacks, and specifically by malware. Existing solutions, consisting of the widely-used antivirus (AV) software, fail to detect newly created and unknown-malware; moreover, by the time the AV is updated, the organization has already been attacked. In this paper, we present a during run-time analysis methodology for a trusted detection of unknown malware on virtual machines (VMs). We conducted trusted analysis of volatile memory dumps taken from a VM and focused on analyzing their system-calls using a sequential-mining-method. We leveraged the most informative system-calls by machine-learning algorithms for the efficient detection of malware in widely used VMs within organizations (i.e. IIS and Email server). We evaluated our methodology in a comprehensive set of experiments over a collections of real-world, advanced, and notorious malware (both ransomware and RAT), and legitimate programs. The results show that our suggested methodology is able to detect the presence of unknown malware, in an average of 97.9% TPR and 0% FPR. Such results and capabilities can form the ground for the development of practical detection-tools for both corporates and companies.

[1]  Sachin Lodha,et al.  POSTER: Locally Virtualized Environment for Mitigating Ransomware Threat , 2016, CCS.

[2]  Pavol Zavarsky,et al.  Comparative Analysis of Volatile Memory Forensics: Live Response vs. Memory Imaging , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[3]  Igor Santos,et al.  Opcode sequences as representation of executables for data-mining-based unknown malware detection , 2013, Inf. Sci..

[4]  Subasish Mohapatra,et al.  Virtualization: A Survey on Concepts, Taxonomy and Associated Security Issues , 2010, 2010 Second International Conference on Computer and Network Technology.

[5]  Brendan Dolan-Gavitt,et al.  Forensic analysis of the Windows registry in memory , 2008, Digit. Investig..

[6]  Vangipuram Radhakrishna,et al.  A Novel Similar Temporal System Call Pattern Mining for Efficient Intrusion Detection , 2016, J. Univers. Comput. Sci..

[7]  Jaeyeon Moon,et al.  Ransomware Analysis and Method for Minimize the Damage , 2016 .

[8]  Alvin Huseinovic,et al.  Virtual machine memory forensics , 2013, 2013 21st Telecommunications Forum Telfor (TELFOR).

[9]  Barak A. Pearlmutter,et al.  Detecting intrusions using system calls: alternative data models , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[10]  S. Dija,et al.  Extraction of memory forensic artifacts from windows 7 RAM image , 2013, 2013 IEEE CONFERENCE ON INFORMATION AND COMMUNICATION TECHNOLOGIES.

[11]  Johannes M. Bauer,et al.  Cybersecurity: Stakeholder incentives, externalities, and policy options , 2009 .

[12]  W. Alink,et al.  Forensic memory analysis: Files mapped in memory , 2008, Digit. Investig..

[13]  Ali Reza Arasteh,et al.  Forensic memory analysis: From stack and code to execution history , 2007, Digit. Investig..

[14]  Yanfang Ye,et al.  IMDS: intelligent malware detection system , 2007, KDD '07.

[15]  Curtis B. Storlie,et al.  Graph-based malware detection using dynamic analysis , 2011, Journal in Computer Virology.

[16]  Felix C. Freiling,et al.  Toward Automated Dynamic Malware Analysis Using CWSandbox , 2007, IEEE Secur. Priv..

[17]  Jiankun Hu,et al.  A Semantic Approach to Host-Based Intrusion Detection Systems Using Contiguousand Discontiguous System Call Patterns , 2014, IEEE Transactions on Computers.

[18]  Andreas Schuster,et al.  Searching for processes and threads in Microsoft Windows memory dumps , 2006, Digit. Investig..

[19]  Zhang Lei,et al.  Memory dump and forensic analysis based on virtual machine , 2014, 2014 IEEE International Conference on Mechatronics and Automation.

[20]  Qiming Chen,et al.  PrefixSpan,: mining sequential patterns efficiently by prefix-projected pattern growth , 2001, Proceedings 17th International Conference on Data Engineering.

[21]  Yuval Elovici,et al.  Malicious Code Detection and Acquisition Using Active Learning , 2007, 2007 IEEE Intelligence and Security Informatics.

[22]  Steve R. White,et al.  Anatomy of a Commercial-Grade Immune System , 1999 .

[23]  Lior Rokach,et al.  ALDROID: efficient update of Android anti-virus software using designated active learning methods , 2016, Knowledge and Information Systems.

[24]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[25]  Lior Rokach,et al.  SFEM: Structural feature extraction methodology for the detection of malicious office documents using machine learning methods , 2016, Expert Syst. Appl..

[26]  Yuval Elovici,et al.  ALDOCX: Detection of Unknown Malicious Microsoft Office Documents Using Designated Active Learning Methods Based on New Structural Feature Extraction Methodology , 2017, IEEE Transactions on Information Forensics and Security.

[27]  Vinay Avasthi,et al.  Ransomware Digital Extortion: A Rising New Age Threat , 2016 .

[28]  Raouf Boutaba,et al.  Cloud computing: state-of-the-art and research challenges , 2010, Journal of Internet Services and Applications.

[29]  Lianhai Wang,et al.  Extracting windows registry information from physical memory , 2011, 2011 3rd International Conference on Computer Research and Development.

[30]  Jie He,et al.  Analyzing Malware by Abstracting the Frequent Itemsets in API Call Sequences , 2013, 2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications.

[31]  Guofei Gu,et al.  Shadow attacks: automatically evading system-call-behavior based malware detection , 2011, Journal in Computer Virology.

[32]  Jesse D. Kornblum.  Exploiting the Rootkit Paradox with Windows Memory Analysis , 2006, Int. J. Digit. EVid..

[33]  Patrick Traynor,et al.  CryptoLock (and Drop It): Stopping Ransomware Attacks on User Data , 2016, 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS).

[34]  Somesh Jha,et al.  Testing malware detectors , 2004, ISSTA '04.

[35]  Engin Kirda,et al.  UNVEIL: A large-scale, automated approach to detecting ransomware (keynote) , 2016, SANER.

[36]  Yuval Elovici,et al.  Malicious Code Detection Using Active Learning , 2009, PinKDD.

[37]  James S. Okolica,et al.  Windows operating systems agnostic memory analysis , 2010 .

[38]  Scott Lowe,et al.  VMware vSphere Design , 2011 .

[39]  Brendan Dolan-Gavitt,et al.  The VAD tree: A process-eye view of physical memory , 2007, Digit. Investig..

[40]  Yuval Shahar,et al.  Classification-driven temporal discretization of multivariate time series , 2014, Data Mining and Knowledge Discovery.

[41]  Yuval Elovici,et al.  Keeping pace with the creation of new malicious PDF files using an active-learning based detection framework , 2016, Security Informatics.

[42]  Craig Valli,et al.  Malware Forensics: Discovery of the Intent of Deception , 2010, J. Digit. Forensics Secur. Law.

[43]  Yuval Shahar,et al.  Inter-labeler and intra-labeler variability of condition severity classification models using active and passive learning methods , 2017, Artif. Intell. Medicine.