Position paper: cloud-based performance testing: issues and challenges

Conducting performance testing is essential to evaluate system performance. With the emergence of cloud computing, applying cloud resources for large-scale performance testing become very attractive. Many organizations have applied cloud-based performance testing in realistic projects. Cloud computing brings many benefits for performance testing, while we also have to face many new problems such as performance variation of cloud platform and security problems. In this overview, we discuss the differences between traditional and cloud-based performance testing. We investigate the state-of-art of cloud-based performance testing. We address the key issues with relevant challenges. For some of the issues, we formalize the problems and give our initial idea. We focus on the quality of workload generation and present our experimental results to validate the existence and degree of the challenges. We think that it is beneficial to apply cloud-based performance testing in many cases.

[1]  Ravi S. Sandhu,et al.  The NIST model for role-based access control: towards a unified standard , 2000, RBAC '00.

[2]  Shau-Yin Tseng,et al.  Integrated design of AES (Advanced Encryption Standard) encrypter and decrypter , 2002, Proceedings IEEE International Conference on Application- Specific Systems, Architectures, and Processors.

[3]  Heiko Ludwig,et al.  Web Service Level Agreement (WSLA) Language Specification , 2003 .

[4]  Wolfgang Emmerich,et al.  SLAng: a language for defining service level agreements , 2003, The Ninth IEEE Workshop on Future Trends of Distributed Computing Systems, 2003. FTDCS 2003. Proceedings..

[5]  Virgílio A. F. Almeida,et al.  Performance by Design - Computer Capacity Planning By Example , 2004 .

[6]  Adrian Paschke.  RBSLA A declarative Rule-based Service Level Agreement Language based on RuleML , 2005, International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA-IAWTIC'06).

[7]  Samuel Kounev,et al.  Performance Modeling and Evaluation of Distributed Component-Based Systems Using Queueing Petri Nets , 2006, IEEE Transactions on Software Engineering.

[8]  Carl Kesselman,et al.  A provisioning model and its comparison with best-effort for performance-cost optimization in grids , 2007, HPDC '07.

[9]  Asser N. Tantawi,et al.  CPU demand for web serving: Measurement analysis and dynamic estimation , 2008, Perform. Evaluation.

[10]  Qi Zhang,et al.  A regression-based analytic model for capacity planning of multi-tier applications , 2008, Cluster Computing.

[11]  Evgenia Smirni,et al.  Anomaly? application change? or workload change? towards automated detection of application performance anomaly and change , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[12]  M. Livny,et al.  The cost of doing science on the cloud: The Montage example , 2008, 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis.

[13]  Sungyoung Lee,et al.  Adoption issues for cloud computing , 2009, MoMM.

[14]  Zohar Ganon,et al.  Cloud-based Performance Testing of Network Management Systems , 2009, 2009 IEEE 14th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks.

[15]  Mingqing Xiao,et al.  The future of automatic test system (ATS) brought by Cloud Computing , 2009, 2009 IEEE AUTOTESTCON.

[16]  Pankesh Patel,et al.  Service Level Agreement in Cloud Computing , 2009 .

[17]  Jorge-Arnulfo Quiané-Ruiz,et al.  Runtime measurements in the cloud , 2010, Proc. VLDB Endow..

[18]  Ossi Taipale,et al.  Software Testing as an Online Service: Observations from Practice , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[19]  George Candea,et al.  Automated software testing as a service , 2010, SoCC '10.

[20]  Antonin Chazalet,et al.  Service Level Checking in the Cloud Computing Context , 2010, 2010 IEEE 3rd International Conference on Cloud Computing.

[21]  K. Popovic,et al.  Cloud computing security issues and challenges , 2010, The 33rd International Convention MIPRO.

[22]  Mitsuhisa Sato,et al.  Large-Scale Software Testing Environment Using Cloud Computing Technology for Dependable Parallel and Distributed Systems , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[23]  Tariq M. King,et al.  Migrating Autonomic Self-Testing to the Cloud , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[24]  Prashant J. Shenoy,et al.  Empirical evaluation of latency-sensitive application performance in the cloud , 2010, MMSys '10.

[25]  Wang Jun,et al.  Software Testing Based on Cloud Computing , 2011, 2011 International Conference on Internet Computing and Information Services.

[26]  O. KuyoroS..  Cloud Computing Security Issues and Challenges , 2011 .

[27]  Ivan Porres,et al.  Towards Automatic Performance and Scalability Testing of Rich Internet Applications in the Cloud , 2011, 2011 37th EUROMICRO Conference on Software Engineering and Advanced Applications.

[28]  Engin Kirda,et al.  A security analysis of Amazon's Elastic Compute Cloud service , 2012, IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN 2012).