Smart CloudBench—A framework for evaluating cloud infrastructure performance

Cloud migration allows organizations to benefit from reduced operational costs, improved flexibility, and greater scalability, and enables them to focus on core business goals. However, it also has the flip side of reduced visibility. Enterprises considering migration of their IT systems to the cloud only have a black box view of the offered infrastructure. While information about server pricing and specification is publicly available, there is limited information about cloud infrastructure performance. Comparison of alternative cloud infrastructure offerings based only on price and server specification is difficult because cloud vendors use heterogeneous hardware resources, offer different server configurations, apply different pricing models and use different virtualization techniques to provision them. Benchmarking the performance of software systems deployed on the top of the black box cloud infrastructure offers one way to evaluate the performance of available cloud server alternatives. However, this process can be complex, time-consuming and expensive, and cloud consumers can greatly benefit from tools that can automate it. Smart CloudBench is a generic framework and system that offers automated, on-demand, real-time and customized benchmarking of software systems deployed on cloud infrastructure. It provides greater visibility and insight into the run-time behavior of cloud infrastructure, helping consumers to compare and contrast available offerings during the initial cloud selection phase, and monitor performance for service quality assurance during the subsequent cloud consumption phase. In this paper, we first discuss the rationale behind our approach for benchmarking the black box cloud infrastructure. Then, we propose a generic architecture for benchmarking representative applications on the heterogeneous cloud infrastructure and describe the Smart CloudBench benchmarking workflow. We also present simple use case scenarios that highlight the need for tools such as Smart CloudBench.

[1]  Babak Falsafi,et al.  Clearing the clouds: a study of emerging scale-out workloads on modern hardware , 2012, ASPLOS XVII.

[2]  Gang Lu,et al.  CloudRank-D: benchmarking and ranking cloud computing systems for data processing applications , 2012, Frontiers of Computer Science.

[3]  Srikanth Kandula,et al.  CloudProphet: towards application performance prediction in cloud , 2011, SIGCOMM 2011.

[4]  Alexandru Iosup,et al.  Benchmarking in the Cloud: What It Should, Can, and Cannot Be , 2012, TPCTC.

[5]  Ryszard Kowalczyk,et al.  Smart CloudMonitor - Providing Visibility into Performance of Black-Box Clouds , 2014, 2014 IEEE 7th International Conference on Cloud Computing.

[6]  Jacky W. Keung,et al.  Evaluating Cloud Platform Architecture with the CARE Framework , 2010, 2010 Asia Pacific Software Engineering Conference.

[7]  Alexandru Iosup,et al.  IaaS cloud benchmarking: approaches, challenges, and experience , 2013, HotTopiCS '13.

[8]  Ryszard Kowalczyk,et al.  Smart CloudBench -- Automated Performance Benchmarking of the Cloud , 2013, 2013 IEEE Sixth International Conference on Cloud Computing.

[9]  Jerome A. Rolia,et al.  Comparing efficiency and costs of cloud computing models , 2012, 2012 IEEE Network Operations and Management Symposium.

[10]  Xiaowei Yang,et al.  CloudCmp: comparing public cloud providers , 2010, IMC '10.

[11]  Michael H. Kalantar,et al.  Semantically-Rich Composition of Virtual Images , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[12]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[13]  Ryszard Kowalczyk,et al.  Smart Cloud Broker: Finding your home in the clouds , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[14]  Stefan Tai,et al.  What Are You Paying For? Performance Benchmarking for Infrastructure-as-a-Service Offerings , 2011, 2011 IEEE 4th International Conference on Cloud Computing.