Generating Test Sequences to Assess the Performance of Elastic Cloud-Based Systems

Elasticity is one of the main features of cloud-based systems (CBSs), where elastic adaptations, such as those to deal with scaling in or scaling out of computational resources, help meet performance requirements under varying workload. There is an industrial need to find configurations of elastic adaptations and workload that could lead to degradation of performance in a CBS, serving possibly millions of users. However, the potentially great number of such configurations poses a challenge: executing and verifying all of them on the cloud can be prohibitively expensive in both, time and cost. We present an approach to model elasticity adaptation due to workload changes as a classification tree model and consequently generate short test sequences of configurations that cover all T-wise interactions between parameters in the model. These test sequences, when executed, help us assess the performance of elastic CBS. Using MongoDB as a case study, test sequences generated by our approach reveal several significant performance degradations.

[1]  Schahram Dustdar,et al.  Testing Elastic Computing Systems , 2013, IEEE Internet Computing.

[2]  Moustafa Ghanem,et al.  Lightweight Resource Scaling for Cloud Applications , 2012, 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012).

[3]  Arnab Sarkar,et al.  Evaluating Reconfiguration Impact in Self-Adaptive Systems -- An Approach Based on Combinatorial Interaction Testing , 2015, 2015 41st Euromicro Conference on Software Engineering and Advanced Applications.

[4]  Thomas Ledoux,et al.  Experimental Analysis on Autonomic Strategies for Cloud Elasticity , 2015, 2015 International Conference on Cloud and Autonomic Computing.

[5]  Kristina Chodorow,et al.  MongoDB - The Definitive Guide: Powerful and Scalable Data Storage , 2019 .

[6]  Gerson Sunyé,et al.  Controlling the elasticity of web applications on cloud computing , 2016, SAC.

[7]  Daniel Moldovan,et al.  SYBL: An Extensible Language for Controlling Elasticity in Cloud Applications , 2013, 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing.

[8]  Kevin Lee,et al.  How a consumer can measure elasticity for cloud platforms , 2012, ICPE '12.

[9]  Samuel Kounev,et al.  Elasticity in Cloud Computing: What It Is, and What It Is Not , 2013, ICAC.

[10]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[11]  Schahram Dustdar,et al.  Testing elastic systems with surrogate models , 2013, 2013 1st International Workshop on Combining Modelling and Search-Based Software Engineering (CMSBSE).

[12]  Divyakant Agrawal,et al.  Albatross: Lightweight Elasticity in Shared Storage Databases for the Cloud using Live Data Migration , 2011, Proc. VLDB Endow..

[13]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[14]  Schahram Dustdar,et al.  Automated testing of cloud-based elastic systems with AUToCLES , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[15]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[16]  Joachim Wegener,et al.  Test Sequence Generation from Classification Trees , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[17]  Jacques Klein,et al.  Pairwise testing for software product lines: comparison of two approaches , 2012, Software Quality Journal.

[18]  J. Wegener,et al.  Test Case Design by Means of the CTE XL , 2000 .

[19]  Daniel Moldovan,et al.  CoMoT -- A Platform-as-a-Service for Elasticity in the Cloud , 2014, 2014 IEEE International Conference on Cloud Engineering.

[20]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[21]  Calton Pu,et al.  Automated control for elastic n-tier workloads based on empirical modeling , 2011, ICAC '11.

[22]  Arnaud Gotlieb,et al.  PACOGEN: Automatic Generation of Pairwise Test Configurations from Feature Models , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.