On the DMA mapping problem in direct device assignment

I/O intensive workloads running in virtual machines can suffer massive performance degradation. Direct assignment of I/O devices to virtual machines is the best performing I/O virtualization mechanism, but its performance still remains far from the bare-metal (non-virtualized) case. The primary gap between direct assignment I/O performance and bare-metal I/O performance is the overhead of mapping the VM's memory pages for DMA in IOMMU translation tables. One could avoid this overhead by mapping all of the VM's pages for the lifetime of the VM, but this leads to memory consumption which is unacceptable in many scenarios. The DMA mapping problem can be stated briefly as "when should a memory page be mapped or unmapped for DMA?" We begin by presenting a theoretical framework for reasoning about the DMA mapping problem. Then, using a quota-based approach, we propose the on-demand DMA mapping strategy, which provides the best DMA mapping performance for a given amount of memory consumed. In particular, on-demand mapping can achieve the same performance as state-of-the-art mapping strategies while consuming much less memory (exact amount depends on the workload's requirements). We present the design and implementation of on-demand mapping in the Linux-based KVM hypervisor and an experimental evaluation of its application to various workloads.