vProbe: Scheduling Virtual Machines on NUMA Systems

With the development of multi-core platforms andcloud computing, Non-Uniform Memory Access (NUMA) architecturehas been dominant in cloud data centers in recentyears. However, NUMA architecture is not well supported invirtualized environments. Because of the semantic gap introducedby the virtualization layer, hypervisors know little aboutthe characteristics of applications running in virtual machines (VMs). More importantly, in order to guarantee hypervisors' applicability, load balance strategies of virtual CPU (VCPU) schedulers do not consider the memory access characteristicsof applications running in VMs, which probably introducessignificant shared resource contention and unnecessary remotememory accesses. In this paper, we propose a NUMA-aware VCPU schedulerbased on Xen, named vProbe, to improve the performanceof memory-intensive applications while maintaining the transparencyof the virtualization layer in NUMA-based servers. It collects performance monitoring units (PMU) data for eachVCPU and analyzes their memory access characteristics. Then, according to the memory access characteristics of each VCPU, it periodically reassigns all memory-intensive VCPUs to eachNUMA node evenly while preferentially allocating them to theirlocal nodes, which aims to alleviate shared resource contentionand reduce unnecessary remote memory accesses. Moreover, when a physical CPU (PCPU) becomes idle, it preferentiallysteals a VCPU from the run queues of PCPUs in the local nodeto this PCPU, which helps to maintain balanced last-level cache (LLC) contention and reduce extra remote memory accesses. Our evaluation shows that vProbe can significantly improve theperformance of memory-intensive applications (e.g., up to 45.2% performance improvement compared with the Credit scheduler) while introducing negligible overheads.

[1]  Shin Gyu Kim,et al.  Virtual machine scheduling for multicores considering effects of shared on-chip last level cache interference , 2012, 2012 International Green Computing Conference (IGCC).

[2]  Brice Goglin,et al.  Enabling high-performance memory migration for multithreaded applications on LINUX , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[3]  Babak Falsafi,et al.  Scale-out NUMA , 2014, ASPLOS.

[4]  Alexandra Fedorova,et al.  Addressing shared resource contention in multicore processors via scheduling , 2010, ASPLOS XV.

[5]  Amin Vahdat,et al.  Dynamic Scheduling of Virtual Machines Running HPC Workloads in Scientific Grids , 2007, 2009 3rd International Conference on New Technologies, Mobility and Security.

[6]  David H. Bailey,et al.  The NAS parallel benchmarks summary and preliminary results , 1991, Proceedings of the 1991 ACM/IEEE Conference on Supercomputing (Supercomputing '91).

[7]  Karsten Schwan,et al.  vNUMA-mgr: Managing VM memory on NUMA platforms , 2010, 2010 International Conference on High Performance Computing.

[8]  Alexandra Fedorova,et al.  A case for NUMA-aware contention management on multicore systems , 2010, 2010 19th International Conference on Parallel Architectures and Compilation Techniques (PACT).

[9]  Vivien Quéma,et al.  MemProf: A Memory Profiler for NUMA Multicore Systems , 2012, USENIX Annual Technical Conference.

[10]  Michael Stumm,et al.  Thread clustering: sharing-aware scheduling on SMP-CMP-SMT multiprocessors , 2007, EuroSys '07.

[11]  Tao Li,et al.  Optimizing virtual machine consolidation performance on NUMA server architecture for cloud workloads , 2014, 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA).

[12]  Torsten Hoefler,et al.  NUMA-aware shared-memory collective communication for MPI , 2013, HPDC.

[13]  Miltos Petridis,et al.  Dynamic Scheduling of Virtual Machines Running HPC Workloads in Scientific Grids , 2009, 2009 3rd International Conference on New Technologies, Mobility and Security.

[14]  Wolfgang Lehner,et al.  ERIS: A NUMA-Aware In-Memory Storage Engine for Analytical Workload , 2014, ADMS@VLDB.

[15]  Aamer Jaleel,et al.  CRUISE: cache replacement and utility-aware scheduling , 2012, ASPLOS XVII.

[16]  Wolfgang Lehner,et al.  BUZZARD: a NUMA-aware in-memory indexing system , 2013, SIGMOD '13.

[17]  Khaled Z. Ibrahim,et al.  The Case for Partitioning Virtual Machines on Multicore Architectures , 2014, IEEE Transactions on Parallel and Distributed Systems.

[18]  Vivien Quéma,et al.  Traffic management: a holistic approach to memory placement on NUMA systems , 2013, ASPLOS '13.

[19]  Kun Wang,et al.  Optimizing virtual machine scheduling in NUMA multicore systems , 2013, 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA).

[20]  Godmar Back,et al.  Perfctr-Xen: a framework for performance counter virtualization , 2011, VEE '11.

[21]  Thomas R. Gross,et al.  Memory management in NUMA multicore systems: trapped between cache contention and interconnect overhead , 2011, ISMM '11.

[22]  Kirk W. Cameron,et al.  Critical path-based thread placement for NUMA systems , 2011, PMBS '11.