Testing and evaluation framework for virtualization technologies

Virtualization technology has attracted considerable attention in recent years. With the increasing popularity of virtualization technology, how to evaluate and ensure its correctness and quality becomes a critical issue. Several methods are proposed to test and evaluate functionalities, performance and quality of different virtualization technologies. However, the testing and evaluation tasks are still complex and time consuming. In this paper, a testing and evaluation framework for virtualization technologies is introduced. The framework aims to provide an integrated environment for software testers from test environment configuration, usage scenario deployment to automatic test and evaluation execution that helps to achieve the functional and quality evaluation of virtualization technologies efficiently. Furthermore, the testing and evaluation framework is built by several well-known open source software, which decreases the deployment cost and retains certain flexibility for further extension.

[1]  Mahidhar Tatineni,et al.  SR-IOV: Performance Benefits for Virtualized Interconnects , 2014, XSEDE '14.

[2]  Richard McDougall,et al.  Virtualization performance: perspectives and challenges ahead , 2010, OPSR.

[3]  Stefano Avallone,et al.  On the Evaluation of VM Provisioning Time in Cloud Platforms for Mission-Critical Infrastructures , 2014, 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[4]  Prashant J. Shenoy,et al.  Empirical evaluation of latency-sensitive application performance in the cloud , 2010, MMSys '10.

[5]  Chia Hung Kao,et al.  A prototype system for object management in private cloud , 2011, 2011 International Conference on Cloud and Service Computing.

[6]  Jacky W. Keung,et al.  Evaluating Cloud Platform Architecture with the CARE Framework , 2010, 2010 Asia Pacific Software Engineering Conference.

[7]  Ramin Nasiri,et al.  A case study for a novel framework for cloud testing , 2014, 2014 11th International Conference on Electronics, Computer and Computation (ICECCO).

[8]  Hao Jiang,et al.  A quantitative study of virtual machine live migration , 2013, CAC.

[9]  Pascal Bouvry,et al.  HPC Performance and Energy-Efficiency of Xen, KVM and VMware Hypervisors , 2013, 2013 25th International Symposium on Computer Architecture and High Performance Computing.

[10]  Alexandru Iosup,et al.  GRENCHMARK: A Framework for Analyzing, Testing, and Comparing Grids , 2006, Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID'06).

[11]  Devarshi Ghoshal,et al.  I/O performance of virtualized cloud environments , 2011, DataCloud-SC '11.

[12]  Raouf Boutaba,et al.  Cloud computing: state-of-the-art and research challenges , 2010, Journal of Internet Services and Applications.

[13]  Ravishankar K. Iyer,et al.  CloudVal: A framework for validation of virtualization environment in cloud infrastructure , 2011, 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN).

[14]  Hasan Sözer,et al.  A Survey of Software Testing in the Cloud , 2012, 2012 IEEE Sixth International Conference on Software Security and Reliability Companion.

[15]  Petter Svärd,et al.  Principles and Performance Characteristics of Algorithms for Live VM Migration , 2015, OPSR.

[16]  Albert Y. Zomaya,et al.  Virt-B: Towards Performance Benchmarking of Virtual Machine Systems , 2014, IEEE Internet Computing.

[17]  Lakshmi Rajamani,et al.  Performance evaluation of Operating Systems in the private cloud with XenServer hypervisor using SIGAR Framework , 2014, 2014 9th International Conference on Computer Science & Education.

[18]  Pavol Zavarsky,et al.  Performance Analysis of Oracle Database in Virtual Environments , 2012, 2012 26th International Conference on Advanced Information Networking and Applications Workshops.

[19]  Jens Grabowski,et al.  A Testing Framework for Assessing Grid and Cloud Infrastructure Interoperability , 2011 .

[20]  Chia Hung Kao,et al.  Performance Testing Framework for REST-Based Web Applications , 2013, 2013 13th International Conference on Quality Software.

[21]  Alexandru Iosup,et al.  C-Meter: A Framework for Performance Analysis of Computing Clouds , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[22]  Yong Yu,et al.  A Synthetical Performance Evaluation of OpenVZ, Xen and KVM , 2010, 2010 IEEE Asia-Pacific Services Computing Conference.

[23]  Salim Hariri,et al.  Proceedings of the 2013 ACM Cloud and Autonomic Computing Conference , 2013 .

[24]  Puneet Sharma,et al.  Framework for testing cloud platforms and infrastructures , 2011, 2011 International Conference on Cloud and Service Computing.

[25]  Yuen-Tak Yu,et al.  Using the 5W+1H Model in Reporting Systematic Literature Review: A Case Study on Software Testing for Cloud Computing , 2013, 2013 13th International Conference on Quality Software.

[26]  Calton Pu,et al.  Expertus: A Generator Approach to Automate Performance Testing in IaaS Clouds , 2012, 2012 IEEE Fifth International Conference on Cloud Computing.

[27]  Kang G. Shin,et al.  Performance Evaluation of Virtualization Technologies for Server Consolidation , 2007 .

[28]  H. Reza Taheri,et al.  Benchmarking a virtualization platform , 2014, 2014 IEEE International Symposium on Workload Characterization (IISWC).

[29]  Ninghui Sun,et al.  Evaluation framework of virtualization systems for cloud computing , 2012, 2012 IEEE Asia Pacific Cloud Computing Congress (APCloudCC).