Volatile memory analysis using the MinHash method for efficient and secured detection of malware in private cloud

Abstract Today, most organizations employ cloud computing environments for both computational reasons and for storing their critical files and data. Virtual servers are an example of widely used virtual resources provided by cloud computing architecture. Therefore, virtual servers are considered an attractive target for cyber-attackers, who launch their attacks by malware such as the well-known remote access trojans (RATs) and more modern malware such as ransomware and cryptojacking. Existing security solutions implemented on virtual servers fail to detect these newly created malware (zero-day attacks). In fact, by the time the security solution is updated, the organization has likely already been attacked. In this study, we present a designated framework aimed at trusted and secured detection of newly created and unknown instances of malware on virtual machines in an organization's private cloud. We took volatile memory dumps from a virtual machine (VM) in a secured and trusted manner, and analyzed all of the data within the memory dumps using the MinHash method; MinHash is well suited for the accurate detection of malware in VMs based on efficient volatile memory dump comparisons. The proposed framework is evaluated in a comprehensive set of experiments of increasing difficulty in which we also measured the detection performance of different classifiers (both similarity and machine learning-based classifiers, using collections of real-world, professional, notorious malware and legitimate applications. The evaluation results show that our framework can detect the anomalous state of a virtual server, as well as known, new, and unknown malware, with very high TPRs (100% for ransomware and RATs) and very low FPRs (1.8% for ransomware and no FPR for RATs). We also show how the methodology's performance can be improved, in terms of required time and storage space, saving more than 86% of these resources. Finally, we demonstrate the generalization capabilities and practicality of our methodology by using transfer learning and learning from just one virtual server in order to detect unknown malware on a different virtual server.