Cloud Computing Benchmarking: A Survey

Cloud computing gives service-oriented access to computing, storage and networking resource. Often, these resources are virtualized. The prospect of being able to scale computing resources to meet user demand has clearly caught the attention of developers and organizational IT leaders over the recent years. Considering the number of cloud computing providers and the different services each provider offers, cloud users need benchmark information that specifically addresses the unique properties of the cloud computing environment such as dynamic scaling. This paper compares five prominent tools (CloudCmp, CloudStone, HiBench, YCSB, and CloudSuite) that present workloads and/or methods for quantitatively comparing cloud computing offerings.

[1]  David A. Patterson,et al.  Computer Architecture: A Quantitative Approach , 1969 .

[2]  Robert L. Grossman,et al.  The Case for Cloud Computing , 2009, IT Professional.

[3]  George Candea,et al.  Cloud9: a software testing service , 2010, OPSR.

[4]  Hans-Arno Jacobsen,et al.  PNUTS: Yahoo!'s hosted data serving platform , 2008, Proc. VLDB Endow..

[5]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[6]  Alexandru Iosup,et al.  Benchmarking in the Cloud: What It Should, Can, and Cannot Be , 2012, TPCTC.

[7]  A. Fox,et al.  Cloudstone : Multi-Platform , Multi-Language Benchmark and Measurement Tools for Web 2 . 0 , 2008 .

[8]  P. Mell,et al.  The NIST Definition of Cloud Computing , 2011 .

[9]  Lieven Eeckhout,et al.  Performance Evaluation and Benchmarking , 2005 .

[10]  Babak Falsafi,et al.  Clearing the clouds: a study of emerging scale-out workloads on modern hardware , 2012, ASPLOS XVII.

[11]  GhemawatSanjay,et al.  The Google file system , 2003 .

[12]  Karl Huppler,et al.  The Art of Building a Good Benchmark , 2009, TPCTC.

[13]  Jie Huang,et al.  The HiBench benchmark suite: Characterization of the MapReduce-based data analysis , 2010, 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010).

[14]  Yong Zhao,et al.  Cloud Computing and Grid Computing 360-Degree Compared , 2008, GCE 2008.

[15]  Michael J. Kavis,et al.  Architecting the Cloud: Design Decisions for Cloud Computing Service Models (Saas, Paas, and Iaas) , 2014 .

[16]  Carsten Binnig,et al.  How is the weather tomorrow?: towards a benchmark for the cloud , 2009, DBTest '09.

[17]  Xiaowei Yang,et al.  CloudCmp: comparing public cloud providers , 2010, IMC '10.

[18]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[19]  Randy H. Katz,et al.  A view of cloud computing , 2010, CACM.