Hardware-in-the-loop simulation for automated benchmarking of cloud infrastructures

To address the challenge of automated performance benchmarking in virtualized cloud infrastructures, an extensible and adaptable framework called CloudBench has been developed to conduct scalable, controllable, and repeatable experiments in such environments. This paper presents the hardware-in-the-loop simulation technique used in CloudBench, which integrates an efficient discrete-event simulation with the cloud infrastructure under test in a closed feedback control loop. The technique supports the decomposition of complex resource usage patterns and provides a mechanism for statistically multiplexing application requests of varied characteristics to generate realistic and emergent behavior. It also exploits parallelism at multiple levels to improve simulation efficiency, while maintaining temporal and causal relationships with proper synchronization. Our experiments demonstrate that the proposed technique can synthesize complex resource usage behavior for effective cloud performance benchmarking.

[1]  Sally Floyd,et al.  Why we don't know how to simulate the Internet , 1997, WSC '97.

[2]  Dong Seong Kim,et al.  End-to-End Performability Analysis for Infrastructure-as-a-Service Cloud: An Interacting Stochastic Models Approach , 2010, 2010 IEEE 16th Pacific Rim International Symposium on Dependable Computing.

[3]  Dave Cliff,et al.  Effects of Component-Subscription Network Topology on Large-Scale Data Centre Performance Scaling , 2010, 2010 15th IEEE International Conference on Engineering of Complex Computer Systems.

[4]  Markus Vorderwinkler,et al.  Soft-commissioning: hardware-in-the-loop-based verification of controller software , 2000, 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165).

[5]  Zhen Li,et al.  HARDWARE-IN-THE-LOOP SIMULATION , 2004 .

[6]  Calton Pu,et al.  Generating Adaptation Policies for Multi-tier Applications in Consolidated Server Environments , 2008, 2008 International Conference on Autonomic Computing.

[7]  Alexandru Iosup,et al.  C-Meter: A Framework for Performance Analysis of Computing Clouds , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[8]  Radu Prodan,et al.  Using a new event-based simulation framework for investigating resource provisioning in Clouds , 2011, Sci. Program..

[9]  Rajkumar Buyya,et al.  CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms , 2011, Softw. Pract. Exp..

[10]  Richard M. Fujimoto,et al.  Exploiting temporal uncertainty in process-oriented distributed simulations , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[11]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[12]  Jesús Carretero,et al.  Design of a New Cloud Computing Simulation Platform , 2011, ICCSA.

[13]  Zoltán Papp,et al.  Distributed hardware-in-the-loop simulator for autonomous continuous dynamical systems with spatially constrained interactions , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[14]  Rajkumar Buyya,et al.  Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters , 2009, HPDC '09.

[15]  Alekh Jindal,et al.  Hadoop++ , 2010 .

[16]  M. Bacic,et al.  On hardware-in-the-loop simulation , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[17]  John Shalf,et al.  Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[18]  Richard M. Fujimoto,et al.  Exploiting temporal uncertainty in parallel and distributed simulations , 1999, Proceedings Thirteenth Workshop on Parallel and Distributed Simulation. PADS 99. (Cat. No.PR00155).

[19]  日本IBMシステムズエンジニアリング株式会社 WebSphere Application Server 開発者ガイド , 2001 .

[20]  Chris Rose,et al.  A Break in the Clouds: Towards a Cloud Definition , 2011 .

[21]  Hosam K. Fathy,et al.  Review of hardware-in-the-loop simulation and its prospects in the automotive area , 2006, SPIE Defense + Commercial Sensing.

[22]  Carsten Binnig,et al.  How is the weather tomorrow?: towards a benchmark for the cloud , 2009, DBTest '09.

[23]  Joseph Idziorek Discrete event simulation model for analysis of horizontal scaling in the cloud computing model , 2010, Proceedings of the 2010 Winter Simulation Conference.

[24]  Xiaohong Jiang,et al.  vTestkit: A Performance Benchmarking Framework for Virtualization Environments , 2010, 2010 Fifth Annual ChinaGrid Conference.

[25]  Archana Ganapathi,et al.  Statistics-driven workload modeling for the Cloud , 2010, 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010).

[26]  Randy H. Katz,et al.  Above the Clouds: A Berkeley View of Cloud Computing , 2009 .

[27]  Calton Pu,et al.  Variations in Performance and Scalability When Migrating n-Tier Applications to Different Clouds , 2011, 2011 IEEE 4th International Conference on Cloud Computing.

[28]  Dorina C. Petriu,et al.  The Future of Software Performance Engineering , 2007, Future of Software Engineering (FOSE '07).

[29]  Li Zhao,et al.  VM3: Measuring, modeling and managing VM shared resources , 2009, Comput. Networks.

[30]  S. Ramey,et al.  Acknowledgement , 2000, NeuroImage.

[31]  Archana Ganapathi,et al.  Towards Understanding Cloud Performance Tradeoffs Using Statistical Workload Analysis and Replay , 2010 .

[32]  Xiaowei Yang,et al.  CloudCmp: comparing public cloud providers , 2010, IMC '10.

[33]  James J. Filliben,et al.  An Efficient Sensitivity Analysis Method for Large Cloud Simulations , 2011, IEEE CLOUD.

[34]  Michael I. Jordan,et al.  Characterizing, modeling, and generating workload spikes for stateful services , 2010, SoCC '10.

[35]  Ferat Sahin,et al.  A discrete event XML based system of systems hardware-in-the-loop simulation for robust threat detection , 2009, 2009 IEEE International Conference on System of Systems Engineering (SoSE).

[36]  Dejan S. Milojicic,et al.  SLA Decomposition: Translating Service Level Objectives to System Level Thresholds , 2007, Fourth International Conference on Autonomic Computing (ICAC'07).