Xen and co.: communication-aware CPU scheduling for consolidated xen-based hosting platforms

Recent advances in software and architectural support for server virtualization have created interest in using this technology in the design of consolidated hosting platforms. Since virtualization enables easier and faster application migration as well as secure co-location of antagonistic applications, higher degrees of server consolidation are likely to result in such virtualization-based hosting platforms (VHPs). We identify a key shortcoming in existing virtual machine monitors (VMMs) that proves to be an obstacle in operating hosting platforms, such as Internet data centers, under conditions of such high consolidation: CPU schedulers that are agnostic to the communication behavior of modern, multi-tier applications. We develop a new communication-aware CPU scheduling algorithm to alleviate this problem. We implement our algorithm in the Xen VMM and build a prototype VHP on a cluster of servers. Our experimental evaluation with realistic Internet server applications and benchmarks demonstrates the performance/cost benefits and the wide applicability of our algorithms. For example, the TPC-W benchmark exhibited improvements in average response times of up to 35% for a variety of consolidation scenarios. A streaming media server hosted on our prototype VHP was able to satisfactorily service up to 3.5 times as many clients as one running on the default Xen.

[1]  Kenneth J. Duda,et al.  Borrowed-virtual-time (BVT) scheduling: supporting latency-sensitive threads in a general-purpose scheduler , 2000, OPSR.

[2]  Mary K. Vernon,et al.  The performance of multiprogrammed multiprocessor scheduling algorithms , 1990, SIGMETRICS '90.

[3]  Chita R. Das,et al.  A closer look at coscheduling approaches for a network of workstations , 1999, SPAA '99.

[4]  Amin Vahdat,et al.  Enforcing Performance Isolation Across Virtual Machines in Xen , 2006, Middleware.

[5]  Allen D. Malony,et al.  Portable profiling and tracing for parallel, scientific applications using C++ , 1998, SPDT '98.

[6]  Daniel A. Menascé,et al.  Resource Allocation for Autonomic Data Centers using Analytic Performance Models , 2005, Second International Conference on Autonomic Computing (ICAC'05).

[7]  Prashant J. Shenoy,et al.  Resource overbooking and application profiling in shared hosting platforms , 2002, OSDI '02.

[8]  Lance M. Berc,et al.  Continuous profiling: where have all the cycles gone? , 1997, TOCS.

[9]  Peter Druschel,et al.  Resource containers: a new facility for resource management in server systems , 1999, OSDI '99.

[10]  Wayne D. Smith,et al.  TPC-W: Benchmarking An Ecommerce Solution , 2001 .

[11]  Patrick Sobalvarro,et al.  Demand-Based Coscheduling of Parallel Jobs on Multiprogrammed Multiprocessors , 1995, JSSPP.

[12]  Marianne Shaw,et al.  Scale and performance in the Denali isolation kernel , 2002, OSDI '02.

[13]  Arnold L. Rosenberg,et al.  Optimal sharing of bags of tasks in heterogeneous clusters , 2003, SPAA '03.

[14]  Mendel Rosenblum,et al.  Cellular disco: resource management using virtual clusters on shared-memory multiprocessors , 2000, TOCS.

[15]  Robbert van Renesse,et al.  JiST: an efficient approach to simulation using virtual machines: Research Articles , 2005 .

[16]  Peter J. Keleher,et al.  Locality and performance of page- and object-based DSMs , 1998, Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing.

[17]  Prashant J. Shenoy,et al.  Surplus fair scheduling: a proportional-share CPU scheduling algorithm for symmetric multiprocessors , 2000, OSDI.

[18]  Andrew Warfield,et al.  Live migration of virtual machines , 2005, NSDI.

[19]  Jason Nieh,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation , 2022 .

[20]  David R. Cheriton,et al.  Borrowed-virtual-time (BVT) scheduling: supporting latency-sensitive threads in a general-purpose scheduler , 1999, OPSR.

[21]  Anand Sivasubramaniam,et al.  A simulation-based study of scheduling mechanisms for a dynamic cluster environment , 2000, ICS '00.

[22]  Andrea C. Arpaci-Dusseau,et al.  Extending Proportional-Share Scheduling to a Network of Workstation , 1997, PDPTA.

[23]  Sriram Govindan,et al.  Communication-aware CPU Management for Consolidated Virtualization-based Hosting Platforms , 2006 .

[24]  Asser N. Tantawi,et al.  An analytical model for multi-tier internet services and its applications , 2005, SIGMETRICS '05.

[25]  Tao Yang,et al.  Integrated resource management for cluster-based Internet services , 2002, OSDI.

[26]  Alan L. Cox,et al.  Optimizing network virtualization in Xen , 2006 .

[27]  Anand Sivasubramaniam,et al.  Managing server energy and operational costs in hosting centers , 2005, SIGMETRICS '05.

[28]  Anand Sivasubramaniam,et al.  Consolidating clients on back-end servers with co-location and frequency control , 2006, SIGMETRICS '06/Performance '06.

[29]  Jason Nieh,et al.  A SMART scheduler for multimedia applications , 2003, TOCS.

[30]  William E. Weihl,et al.  Lottery scheduling: flexible proportional-share resource management , 1994, OSDI '94.

[31]  Abhishek Chandra,et al.  An observation-based approach towards self-managing Web servers , 2002, IEEE 2002 Tenth IEEE International Workshop on Quality of Service (Cat. No.02EX564).

[32]  Sara Sprenkle,et al.  Managing Mixed-Use Clusters with Cluster-on-Demand , 2002 .

[33]  Wei Jin,et al.  USENIX Association Proceedings of USITS ’ 03 : 4 th USENIX Symposium on Internet Technologies and Systems , 2003 .

[34]  Michael B. Jones,et al.  CPU reservations and time constraints: efficient, predictable scheduling of independent activities , 1997, SOSP.

[35]  Dhabaleswar K. Panda,et al.  High Performance VMM-Bypass I/O in Virtual Machines , 2006, USENIX Annual Technical Conference, General Track.

[36]  WhitakerAndrew,et al.  Scale and performance in the Denali isolation kernel , 2002 .

[37]  Carl A. Waldspurger,et al.  Memory resource management in VMware ESX server , 2002, OSDI '02.

[38]  Daniel A. Menascé,et al.  Autonomic Virtualized Environments , 2006, International Conference on Autonomic and Autonomous Systems (ICAS'06).

[39]  Monica S. Lam,et al.  Optimizing the migration of virtual computers , 2002, OPSR.

[40]  Ludmila Cherkasova,et al.  XenMon: QoS Monitoring and Performance Profiling Tool , 2005 .

[41]  Andrew Warfield,et al.  Xen and the art of virtualization , 2003, SOSP '03.

[42]  Mark S. Squillante,et al.  Modeling and analysis of dynamic coscheduling in parallel and distributed environments , 2002, SIGMETRICS '02.

[43]  Anand Sivasubramaniam,et al.  Improving parallel job scheduling by combining gang scheduling and backfilling techniques , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[44]  Andrea C. Arpaci-Dusseau,et al.  Implicit coscheduling: coordinated scheduling with implicit information in distributed systems , 2001, TOCS.

[45]  Robert P. Goldberg,et al.  Survey of virtual machine research , 1974, Computer.

[46]  UrgaonkarBhuvan,et al.  Resource overbooking and application profiling in shared hosting platforms , 2002 .