Test case minimization approach using fault detection and combinatorial optimization techniques for configuration-aware structural testing

Abstract This paper presents a technique to minimize the number of test cases in configuration-aware structural testing. Combinatorial optimization is used first to generate an optimized test suite by sampling the input configuration. Second, for further optimization, the generated test suite is filtered based on an adaptive mechanism by using a mutation testing technique. The initialized test suite is optimized using cuckoo search (CS) along with combinatorial approach, and mutation testing is used to seed different faults to the software-under-test, as well as to filter the test cases based on the detected faults. To measure the effectiveness of the technique, an empirical study is conducted on a software system. The technique proves its effectiveness through the conducted case study. The paper also shows the application of combinatorial optimization and CS to the software testing.

[1]  Ching-Shui Cheng,et al.  Orthogonal Arrays with Variable Numbers of Symbols , 1980 .

[2]  Myra B. Cohen,et al.  An Improved Meta-heuristic Search for Constrained Interaction Testing , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[3]  Kamal Zuhairi Zamli,et al.  A variable strength interaction test suites generation strategy using Particle Swarm Optimization , 2011, J. Syst. Softw..

[4]  Xiao Qu,et al.  Testing of Configurable Systems , 2013, Adv. Comput..

[5]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[6]  Charles J. Colbourn,et al.  Experimental designs in software engineering: d-optimal designs and covering arrays , 2004, WISER '04.

[7]  Huimin Wang,et al.  Parameter tuning of particle swarm optimization by using Taguchi method and its application to motor design , 2014, 2014 4th IEEE International Conference on Information Science and Technology.

[8]  Sven Apel,et al.  Types and modularity for implicit invocation with implicit announcement , 2010, TSEM.

[9]  Mario Cannataro,et al.  Protein-to-protein interactions: Technologies, databases, and algorithms , 2010, CSUR.

[10]  Ramin Rajabioun,et al.  Cuckoo Optimization Algorithm , 2011, Appl. Soft Comput..

[11]  Myra B. Cohen,et al.  Repairing GUI Test Suites Using a Genetic Algorithm , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[12]  Myra B. Cohen,et al.  Towards incremental adaptive covering arrays , 2007, ESEC-FSE '07.

[13]  Xiang Chen,et al.  Variable Strength Interaction Testing with an Ant Colony System Approach , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[14]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[15]  Hareton K. N. Leung,et al.  The Minimal Failure-Causing Schema of Combinatorial Testing , 2011, TSEM.

[16]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[17]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[18]  Chee Peng Lim,et al.  Application of Particle Swarm Optimization to uniform and variable strength covering array construction , 2012, Appl. Soft Comput..

[19]  Xin-She Yang,et al.  Nature-Inspired Metaheuristic Algorithms , 2008 .

[20]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[21]  Bestoun S. Ahmed,et al.  An efficient strategy for covering array construction with fuzzy logic-based adaptive swarm optimization for software testing use , 2015, Expert Syst. Appl..

[22]  Bestoun S. Ahmed,et al.  Generating combinatorial test cases using Simplified Swarm Optimization (SSO) algorithm for automated GUI functional testing , 2014 .

[23]  Yu Lei,et al.  IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing , 2008 .

[24]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[25]  Xiangtao Li,et al.  Modified cuckoo search algorithm with self adaptive parameter method , 2015, Inf. Sci..

[26]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[27]  Asha Gowda Karegowda,et al.  A Brief Survey on Cuckoo Search Applications , 2014 .

[28]  Charles J. Colbourn,et al.  Covering arrays of higher strength from permutation vectors , 2006 .

[29]  Thang Trung Nguyen,et al.  Modified cuckoo search algorithm for short-term hydrothermal scheduling , 2015 .

[30]  Myra B. Cohen,et al.  Moving Forward with Combinatorial Interaction Testing , 2014, Computer.

[31]  J. Czerwonka Pairwise Testing in Real World Practical Extensions to Test Case Generators , 2006 .

[32]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[33]  Daniel J. Velleman,et al.  Permutations and Combination Locks , 1995 .

[34]  Phil McMinn,et al.  Search-based software test data generation: a survey: Research Articles , 2004 .

[35]  Xin-She Yang,et al.  Engineering optimisation by cuckoo search , 2010 .

[36]  Xin-She Yang,et al.  Cuckoo Search via Lévy flights , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[37]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[38]  Myra B. Cohen,et al.  Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization , 2007, 2007 IEEE International Conference on Software Maintenance.

[39]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[40]  Donald L. Kreher,et al.  On the state of strength‐three covering arrays , 2002 .

[41]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[42]  Shahin Gelareh,et al.  A Survey of Meta-Heuristic Solution Methods for the Quadratic Assignment Problem , 2007 .

[43]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[44]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999 .

[45]  Myra B. Cohen,et al.  Designing Test Suites for Software Interactions Testing , 2004 .

[46]  Thang Trung Nguyen,et al.  Cuckoo search algorithm for short-term hydrothermal scheduling , 2014 .

[47]  Ajith Abraham,et al.  A fuzzy adaptive turbulent particle swarm optimisation , 2007 .

[48]  Charles J. Colbourn,et al.  Merging covering arrays and compressing multiple sequence alignments , 2009, Discret. Appl. Math..

[49]  Z. ZamliKamal,et al.  T-Way Testing Strategies: A Critical Survey and Analysis , 2013 .

[50]  Xin-She Yang,et al.  Discrete cuckoo search algorithm for the travelling salesman problem , 2014, Neural Computing and Applications.

[51]  Victor V. Kuliamin,et al.  A survey of methods for constructing covering arrays , 2011, Programming and Computer Software.

[52]  Seyed Javad Mirabedini,et al.  Combining Cuckoo and Tabu Algorithms for Solving Quadratic Assignment Problems , 2012 .

[53]  Soumen Maity,et al.  Improved test generation algorithms for pair-wise testing , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[54]  Gang Xu,et al.  An adaptive parameter tuning of particle swarm optimization algorithm , 2013, Appl. Math. Comput..

[55]  D. Richard Kuhn,et al.  Combinatorial Testing for Software: an Adaptation of Design of Experiments , 2013 .

[56]  Xin-She Yang,et al.  Cuckoo search: recent advances and applications , 2013, Neural Computing and Applications.

[57]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[58]  Douglas C. Montgomery,et al.  Software performance testing using covering arrays: efficient screening designs with categorical factors , 2005, WOSP '05.