Tracing Privileged Memory Accesses to Discover Software Vulnerabilities

Shared Memory is an important mechanism for efficient inter-process communication. When one side of the communication has higher privileges than its counterpart, the shared memory interface becomes a trust boundary and privileged code operating on it needs to be audited for security vulnerabilities. In this thesis we present an approach based on memory tracing to discover vulnerabilities in shared memory interfaces. In contrast to other works in this area, the presented implementation is based on hardware-assisted virtualization and uses manipulation of EPT permissions to intercept memory accesses. We evaluate our implementation against paravirtualized device drivers for the Xen hypervisor, which use shared memory for inter-domain communication. Besides successfully identifying the privileged components responsible for processing untrusted shared memory data, the presented analysis algorithms are used to discover three novel security vulnerabilities in security critical backend components.