XScope: Memory Introspection Based Malicious Application Detection

The malicious applications may hide itself to evade the detection of security tools placed inside or outside of the system. To detect these hidden applications, existing approaches employ virtual machine introspection to intercept the process switches and identify the hidden process once it is being switched. However, this method imposes heavy overhead due to frequent process switches. Moreover, it fails to detect some malicious applications which hide in the system without execution for a long time. To solve these problems, we propose XScope, a memory introspection based hidden application detection approach. First, XScope categorizes the memory pages of virtual machine (VM) using memory introspection, and identifies the anonymous pages which are related to user applications. Second, XScope extracts the processes by analyzing the anonymous memory pages, to acquire a full map of processes at the virtual machine monitor (VMM) layer. Third, XScope compares the map acquired at the VMM layer against the map acquired in the VM for detecting the hidden applications. We implement a prototype system and conduct a set of experiments. The experimental results show that XScope is able to detect the hidden malicious application without significant overhead.