When eXtended Para - Virtualization (XPV) Meets NUMA

This paper addresses the problem of efficiently virtualizing NUMA architectures. The major challenge comes from the fact that the hypervisor regularly reconfigures the placement of a virtual machine (VM) over the NUMA topology. However, neither guest operating systems (OSes) nor system runtime libraries (e.g., Hotspot) are designed to consider NUMA topology changes at runtime, leading end user applications to unpredictable performance. This paper presents eXtended Para-Virtualization (XPV), a new principle to efficiently virtualize a NUMA architecture. XPV consists in revisiting the interface between the hypervisor and the guest OS, and between the guest OS and system runtime libraries (SRL) so that they can dynamically take into account NUMA topology changes. The paper presents a methodology for systematically adapting legacy hypervisors, OSes, and SRLs. We have applied our approach with less than 2k line of codes in two legacy hypervisors (Xen and KVM), two legacy guest OSes (Linux and FreeBSD), and three legacy SRLs (Hotspot, TCMalloc, and jemalloc). The evaluation results showed that XPV outperforms all existing solutions by up to 304%.

[1]  Kun Wang,et al.  Optimizing virtual machine scheduling in NUMA multicore systems , 2013, 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA).

[2]  Babak Falsafi,et al.  Clearing the clouds: a study of emerging scale-out workloads on modern hardware , 2012, ASPLOS XVII.

[3]  Ludmila Cherkasova,et al.  Measuring CPU Overhead for I/O Processing in the Xen Virtual Machine Monitor , 2005, USENIX ATC, General Track.

[4]  Danny Jones,et al.  VM Live Migration At Scale , 2018, VEE.

[5]  Daniel Hagimont,et al.  Application-specific quantum for multi-core platform scheduler , 2016, EuroSys.

[6]  Hai Jin,et al.  vProbe: Scheduling Virtual Machines on NUMA Systems , 2016, 2016 IEEE International Conference on Cluster Computing (CLUSTER).

[7]  Scott Devine,et al.  Disco: running commodity operating systems on scalable multiprocessors , 1997, TOCS.

[8]  Vivien Quéma,et al.  Thread and Memory Placement on NUMA Systems: Asymmetry Matters , 2015, USENIX Annual Technical Conference.

[9]  Kenneth C. Knowlton,et al.  A fast storage allocator , 1965, CACM.

[10]  Jaehyuk Huh,et al.  The Effect of Multi-core on HPC Applications in Virtualized Systems , 2010, Euro-Par Workshops.

[11]  The CPU Scheduler in VMware vSphere ® 5 , 2013 .

[12]  David Chisnall,et al.  The Definitive Guide to the Xen Hypervisor , 2007 .

[13]  A WaldspurgerCarl.  Memory resource management in VMware ESX server , 2002 .

[14]  Nhan Nguyen,et al.  NumaGiC: a Garbage Collector for Big Data on Big NUMA Machines , 2015, ASPLOS.

[15]  Gerald J. Popek,et al.  Formal requirements for virtualizable third generation architectures , 1974, SOSP '73.

[16]  Daniel Hagimont,et al.  Working Set Size Estimation Techniques in Virtualized Environments , 2018, Proc. ACM Meas. Anal. Comput. Syst..

[17]  Todor Ivanov,et al.  BigBench V2: The New and Improved BigBench , 2017, 2017 IEEE 33rd International Conference on Data Engineering (ICDE).

[18]  Tao Li,et al.  Optimizing virtual machine consolidation performance on NUMA server architecture for cloud workloads , 2014, 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA).

[19]  Vivien Quéma,et al.  Large Pages May Be Harmful on NUMA Systems , 2014, USENIX Annual Technical Conference.

[20]  Pierre Sens,et al.  An interface to implement NUMA policies in the Xen hypervisor , 2017, EuroSys.

[21]  Marc Shapiro,et al.  A study of the scalability of stop-the-world garbage collectors on multicores , 2013, ASPLOS '13.