Performance Interference of Memory Thrashing in Virtualized Cloud Environments: A Study of Consolidated n-Tier Applications

Modern datacenters employ server virtualization and consolidation to reduce the cost of operation and to maximize profit. However, interference among consolidated virtual machines (VMs) has barred mission-critical applications due to unpredictable performance. Through extensive measurements of RUBBoS n-tier benchmark, we found a major source of performance unpredictability: the memory thrashing caused by VM consolidation can reduce the system throughput by 46% although memory was not over-committed. On a physical host with 4 consolidated VMs, we observed two distinct operational modes during a typical RUBBoS benchmark experiment. Over the first half of run-time session we found frequent CPU IOwait causing very long response time requests even though the system is under read-only CPU intensive workload, however, the latter half showed no such CPU abnormalities (IOwait). Using ElbaLens - a lightweight tracing tool, we conducted fine-grain analyses at time granularities as short as 50ms and found that the abnormal IOwait is caused by transient memory thrashing among consolidated VMs. The abnormal IOwait induces queue overflows that propagate through the entire n-tier system, resulting in very long response time requests due to frequent TCP retransmissions. We provide three practical techniques such as VM migration, memory reallocation, soft resource reallocation and show that they can mitigate the effects of performance interference among consolidated VMs.