Performance Evaluation of Cloud Infrastructure using Complex Workloads

Infrastructure as a Service (IaaS) is a delivery model of cloud computing, which provides the ability to users to acquire and release resources according to their demand and pay according to their usage. Resources are provisioned from the cloud as Virtual Machines (VMs), many of which can be deployed on a single computing node, realizing a multi-tenancy model. While virtualization and multi-tenancy are two sources of workload-execution overhead that have been studied in the past, we still need a thorough, empirical investigation of the joint impact of these overheads, on workload execution. Additionally, commercial and private IaaS providers offer mechanisms that facilitate the lease and use of single infrastructure resources, but to execute multi-job workloads IaaS users still need to select adequate provisioning and allocation policies to instantiate resources and map computational jobs to them. Even though some studies on the policies employed in cloud environments already exist, current and potential IaaS users need deeper insight on the achieved performance and incurred cost of the used policies, derived through empirical investigation. In this work, we address these problems with the use of SkyMark, a performance analysis framework for IaaS clouds. SkyMark has three key features: ?rst, it is designed to analyze IaaS deployments through a sequence of automated tests and the subsequent automated analysis of results. Second, it can analyze the impact of individual provisioning and allocation policies to the performance of the workload execution. Lastly, it is able to generate complex workloads, stressing any of the compute, memory and disk components. With the use of SkyMark, we ?rst study the overheads that the cloud software stack imposes to the workload execution. Subsequently, we analyze the performance and cost of six provisioning and three allocation policies through experimentation in three IaaS environments, including Amazon EC2.

[1]  Andrea C. Arpaci-Dusseau,et al.  The interaction of parallel and sequential workloads on a network of workstations , 1995, SIGMETRICS '95/PERFORMANCE '95.

[2]  Sebastien Goasguen,et al.  Dynamic Provisioning of Virtual Organization Clusters , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[3]  Calton Pu,et al.  Understanding Performance Interference of I/O Workload in Virtualized Cloud Environments , 2010, 2010 IEEE 3rd International Conference on Cloud Computing.

[4]  Ashraf Aboulnaga,et al.  Database systems on virtual machines: How much do you lose? , 2008, 2008 IEEE 24th International Conference on Data Engineering Workshop.

[5]  Albert G. Greenberg,et al.  VL2: a scalable and flexible data center network , 2009, SIGCOMM '09.

[6]  Alexandru Iosup,et al.  Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing , 2011, IEEE Transactions on Parallel and Distributed Systems.

[7]  Alexandru Iosup,et al.  A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing , 2009, CloudComp.

[8]  Rajkumar Buyya,et al.  Introduction to Cloud Computing , 2011, CloudCom 2011.

[9]  Wei Lu,et al.  Performing Large Science Experiments on Azure: Pitfalls and Solutions , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[10]  Thomas A. Henzinger,et al.  FlexPRICE: Flexible Provisioning of Resources in a Cloud Environment , 2010, 2010 IEEE 3rd International Conference on Cloud Computing.

[11]  Paul Marshall,et al.  Elastic Site: Using Clouds to Elastically Extend Site Resources , 2010, 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing.

[12]  Alexandru Iosup,et al.  An Analysis of Provisioning and Allocation Policies for Infrastructure-as-a-Service Clouds , 2012, 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012).

[13]  Ludmila Cherkasova,et al.  Measuring CPU Overhead for I/O Processing in the Xen Virtual Machine Monitor , 2005, USENIX ATC, General Track.

[14]  Alexandru Iosup,et al.  The Grid Workloads Archive , 2008, Future Gener. Comput. Syst..

[15]  David J. Lilja,et al.  Measuring computer performance : A practitioner's guide , 2000 .

[16]  M. Prange,et al.  Scientific Computing in the Cloud , 2008, Computing in Science & Engineering.

[17]  Carsten Binnig,et al.  How is the weather tomorrow?: towards a benchmark for the cloud , 2009, DBTest '09.

[18]  Jeffrey S. Vetter,et al.  Xen-Based HPC: A Parallel I/O Perspective , 2008, 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid (CCGRID).

[19]  Benny Rochwerger,et al.  On the Management of Virtual Machines for Cloud Infrastructures , 2011 .

[20]  Richard Wolski,et al.  The Eucalyptus Open-Source Cloud-Computing System , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[21]  Alexander Zeier,et al.  The effects of virtualization on main memory systems , 2010, DaMoN '10.

[22]  Lavanya Ramakrishnan,et al.  Performance and cost analysis of the Supernova factory on the Amazon AWS cloud , 2011, CloudCom 2011.

[23]  Lavanya Ramakrishnan,et al.  Performance and cost analysis of the Supernova factory on the Amazon AWS cloud , 2011, Sci. Program..

[24]  Randy H. Katz,et al.  Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center , 2011, NSDI.

[25]  Alexandru Iosup,et al.  On the Performance Variability of Production Cloud Services , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[26]  Chandra Krintz,et al.  Paravirtualization for HPC Systems , 2006, ISPA Workshops.

[27]  Albert G. Greenberg,et al.  The cost of a cloud: research problems in data center networks , 2008, CCRV.

[28]  Gil Neiger,et al.  Intel virtualization technology , 2005, Computer.

[29]  Xiaowei Yang,et al.  CloudCmp: comparing public cloud providers , 2010, IMC '10.

[30]  Guillaume Pierre,et al.  EC2 Performance Analysis for Resource Provisioning of Service-Oriented Applications , 2009, ICSOC/ServiceWave Workshops.

[31]  Miron Livny,et al.  Condor: a distributed job scheduler , 2001 .

[32]  Francisco Vilar Brasileiro,et al.  Investigating Business-Driven Cloudburst Schedulers for E-Science Bag-of-Tasks Applications , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[33]  Scott Shenker,et al.  Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling , 2010, EuroSys '10.

[34]  Lei Shi,et al.  Dcell: a scalable and fault-tolerant network structure for data centers , 2008, SIGCOMM '08.

[35]  Alexandru Iosup,et al.  ExPERT: Pareto-Efficient Task Replication on Grids and a Cloud , 2012, 2012 IEEE 26th International Parallel and Distributed Processing Symposium.

[36]  Thomas J. Hacker,et al.  Flexible resource allocation for reliable virtual cluster computing systems , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[37]  Rajkumar Buyya,et al.  Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters , 2009, HPDC '09.

[38]  A. Kivity,et al.  kvm : the Linux Virtual Machine Monitor , 2007 .

[39]  William Gropp,et al.  Beowulf Cluster Computing with Linux , 2003 .

[40]  Sornthep Vannarat,et al.  Autonomic resource provisioning in rocks clusters using Eucalyptus cloud computing , 2010, MEDES.

[41]  Edward Walker,et al.  Benchmarking Amazon EC2 for High-Performance Scientific Computing , 2008, login Usenix Mag..

[42]  P. Mell,et al.  The NIST Definition of Cloud Computing , 2011 .

[43]  Borja Sotomayor,et al.  Virtual Infrastructure Management in Private and Hybrid Clouds , 2009, IEEE Internet Computing.

[44]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[45]  Rajkumar Buyya,et al.  Adapting Market-Oriented Scheduling Policies for Cloud Computing , 2010, ICA3PP.

[46]  Tom Killalea.  Meet the Virts , 2008, ACM Queue.

[47]  Jin-Soo Kim,et al.  Cost optimized provisioning of elastic resources for application workflows , 2011, Future Gener. Comput. Syst..

[48]  Alexandru Iosup,et al.  C-Meter: A Framework for Performance Analysis of Computing Clouds , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[49]  Miron Livny,et al.  The cost of doing science on the cloud: The Montage example , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[50]  Füsun Özgüner,et al.  Run-time statistical estimation of task execution times for heterogeneous distributed computing , 1996, Proceedings of 5th IEEE International Symposium on High Performance Distributed Computing.

[51]  Alexandru Iosup,et al.  The performance of bags-of-tasks in large-scale distributed systems , 2008, HPDC '08.

[52]  Carsten Franke,et al.  On Grid Performance Evaluation Using Synthetic Workloads , 2006, JSSPP.

[53]  Jie Li,et al.  Cloud auto-scaling with deadline and budget constraints , 2010, 2010 11th IEEE/ACM International Conference on Grid Computing.

[54]  Marianne Shaw,et al.  Denali: Lightweight Virtual Machines for Distributed and Networked Applications , 2001 .

[55]  Matei Ripeanu,et al.  Amazon S3 for science grids: a viable solution? , 2008, DADC '08.

[56]  Minglu Li,et al.  Dynamic adaptive scheduling for virtual machines , 2011, HPDC '11.

[57]  Julien Gossa,et al.  Cost-Wait Trade-Offs in Client-Side Resource Provisioning with Elastic Clouds , 2011, 2011 IEEE 4th International Conference on Cloud Computing.

[58]  Alexandru Iosup,et al.  Grid Computing Workloads , 2011, IEEE Internet Computing.

[59]  Rajkumar Buyya,et al.  Cloud Computing Principles and Paradigms , 2011 .

[60]  Radu Prodan,et al.  Resource Management for Hybrid Grid and Cloud Computing , 2010, Cloud Computing.

[61]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[62]  Willy Zwaenepoel,et al.  Diagnosing performance overheads in the xen virtual machine environment , 2005, VEE '05.

[63]  Naveen Sharma,et al.  Towards autonomic workload provisioning for enterprise Grids and clouds , 2009, 2009 10th IEEE/ACM International Conference on Grid Computing.

[64]  John Shalf,et al.  Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[65]  Rajkumar Buyya,et al.  SLA-Based Resource Allocation for Software as a Service Provider (SaaS) in Cloud Computing Environments , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[66]  Eli M. Dow,et al.  Xen and the Art of Repeated Research , 2004, USENIX Annual Technical Conference, FREENIX Track.

[67]  Alexandru Iosup,et al.  GRENCHMARK: A Framework for Analyzing, Testing, and Comparing Grids , 2006, Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID'06).

[68]  Toshio Nakatani,et al.  Performance variations of two open-source cloud platforms , 2010, IEEE International Symposium on Workload Characterization (IISWC'10).

[69]  Mohamed El‐Refaey,et al.  Virtual Machines Provisioning and Migration Services , 2011, CloudCom 2011.

[70]  Randy H. Katz,et al.  Above the Clouds: A Berkeley View of Cloud Computing , 2009 .

[71]  Love H. Seawright,et al.  VM/370 - A Study of Multiplicity and Usefulness , 1979, IBM Syst. J..

[72]  John Zahorjan,et al.  Scheduling a mixed interactive and batch workload on a parallel, shared memory supercomputer , 1992, Proceedings Supercomputing '92.

[73]  Jesús Carretero,et al.  Predictive Data Grouping and Placement for Cloud-Based Elastic Server Infrastructures , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[74]  Aman Kansal,et al.  Q-clouds: managing performance interference effects for QoS-aware clouds , 2010, EuroSys '10.