New Methods for Detecting Malware Infections and New Attacks against Hardware Virtualization

In my dissertation, I address problems in two domains: (i) Detection of unknown malware and (ii) finding new attacks against hardware virtualization. Accordingly, this dissertation is divided into two parts. In the first part of the dissertation, I propose Membrane, a memory forensics tool to detect code injection attacks. Instead of trying to detect the code injection event itself, I focus on the changes it causes on the paging behavior of the Windows operating system. As my method focuses on the anomalies caused by code injection in paging events, I am able to detect a wide range of code injection techniques. My results indicate that on Windows XP we can detect malware behavior with 91-98% success. On Windows 7, a good detection rate is maintained except for malware injecting into explorer.exe where the success of detection decreases to 75-86%. My approach can detect stealthy malware attacks, even advanced targeted attacks using code injection. Still in the first part, I propose a new system monitoring framework that can serve as an enabler for automated malware detection on live systems. My approach takes advantage of the increased availability of hardware assisted virtualization capabilities of modern CPUs, and its basic novelty consists in launching a hypervisor layer on the live system without stopping and restarting it. This hypervisor runs at a higher privilege level than the OS itself, thus, it can be used to observe the behavior of the analyzed system in a transparent manner. For this purpose, I also propose a novel system call tracing method that is designed to be configurable in terms of transparency and granularity. In the second part of the dissertation, I shed light on VM related threats and defences by implementing, testing, and categorizing a wide range of known and unknown attacks based on directly assigned devices. I executed these attacks on an exhaustive set of VMM configurations to determine their potential impact. My experiments suggest that most of the previously known attacks are ineffective in current VMM setups. I also developed an automatic tool, called PTFuzz, to discover hardware-level problems that affect current VMMs. By using PTFuzz, I found several cases of unexpected hardware behaviour, and a major vulnerability on Intel platforms that potentially impacts a large set of machines used in the wild. These vulnerabilities affect unprivileged virtual machines that use a directly assigned device (e.g., network card) and have all the existing hardware protection mechanisms enabled. Such vulnerabilities allow either an attacker to generate a host-side interrupt or hardware faults, violating expected isolation properties. These can cause host software (e.g., VMM) halt as well as they might open the door for practical VMM exploitations. I believe that my study can help cloud providers and researchers to better understand the limitations of their current architectures to provide secure hardware virtualization and prepare for future attacks. At the same time, security practitioners make heavy use of various virtualization techniques to create sandboxing environments that provide a certain level of isolation between the host and the code being analysed. However, most of these are easy to be detected and evaded. The introduction of hardware assisted virtualization (Intel VT and AMD-V) made the creation of novel, out-of-the-guest malware analysis platforms possible. These allow for a high level of transparency by residing completely outside the guest operating system being examined, thus conventional in-memory detection scans are ineffective. Furthermore, such analyzers resolve the shortcomings that stem from inaccurate system emulation, in-guest timings, privileged operations and so on. Finally, I introduce novel approaches that make the detection of hardware assisted virtualization plat-