TCA: An Efficient Two-Mode Meta-Heuristic Algorithm for Combinatorial Test Generation (T)

Covering arrays (CAs) are often used as test suites for combinatorial interaction testing to discover interaction faults of real-world systems. Most real-world systems involve constraints, so improving algorithms for covering array generation (CAG) with constraints is beneficial. Two popular methods for constrained CAG are greedy construction and meta-heuristic search. Recently, a meta-heuristic framework called two-mode local search has shown great success in solving classic NPhard problems. We are interested whether this method is also powerful in solving the constrained CAG problem. This work proposes a two-mode meta-heuristic framework for constrained CAG efficiently and presents a new meta-heuristic algorithm called TCA. Experiments show that TCA significantly outperforms state-of-the-art solvers on 3-way constrained CAG. Further experiments demonstrate that TCA also performs much better than its competitors on 2-way constrained CAG.

[1]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[2]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[3]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[4]  Yu-Wen Tung,et al.  Automating test case generation for the new generation mission software system , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[5]  Tatsuhiro Tsuchiya,et al.  A new method for constructing pair-wise covering designs for software testing , 2002, Information Processing Letters.

[6]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[7]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[8]  Moataz A. Ahmed,et al.  Pair-wise test coverage using genetic algorithms , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[9]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[10]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[11]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[12]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[13]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[14]  Chu Min Li,et al.  Diversification and Determinism in Local Search for Satisfiability , 2005, SAT.

[15]  Baowen Xu,et al.  Automatic Test Generation for N-Way Combinatorial Testing , 2005, QoSA/SOQUA.

[16]  Myra B. Cohen,et al.  A framework of greedy methods for constructing interaction test suites , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[17]  Mark Harman,et al.  An empirical study of the robustness of two module clustering fitness functions , 2005, GECCO '05.

[18]  Jonas Mellin,et al.  Handling Constraints in the Input Space when Using Combination Strategies for Software Testing , 2006 .

[19]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[20]  Generating Optimal Test Set for Neighbor Factors Combinatorial Testing , 2006, QSIC.

[21]  Baowen Xu,et al.  A New Heuristic for Test Suite Generation for Pair-wise Testing , 2006, SEKE.

[22]  Charles J. Colbourn,et al.  One-test-at-a-time heuristic search for interaction test suites , 2007, GECCO '07.

[23]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[24]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[25]  Charles J. Colbourn,et al.  The density algorithm for pairwise interaction testing , 2007, Softw. Test. Verification Reliab..

[26]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[27]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[28]  Xu Baowen,et al.  Generating combinatorial test suite for interaction relationship , 2007 .

[29]  Mark Harman,et al.  Pareto optimal search based refactoring at the design level , 2007, GECCO '07.

[30]  Baowen Xu,et al.  Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite , 2008, 2008 The Eighth International Conference on Quality Software.

[31]  Yu Lei,et al.  IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing , 2008 .

[32]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[33]  Charles J. Colbourn,et al.  A density‐based greedy algorithm for higher strength covering arrays , 2009, Softw. Test. Verification Reliab..

[34]  Myra B. Cohen,et al.  An Improved Meta-heuristic Search for Constrained Interaction Testing , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[35]  James D. McCaffrey,et al.  Generation of Pairwise Test Sets Using a Genetic Algorithm , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[36]  Charles J. Colbourn,et al.  A density-based greedy algorithm for higher strength covering arrays , 2009 .

[37]  José Torres-Jiménez,et al.  Construction of Mixed Covering Arrays of Variable Strength Using a Tabu Search Approach , 2010, COCOA.

[38]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[39]  Adrian Balint,et al.  Improving Stochastic Local Search for SAT with a New Probability Distribution , 2010, SAT.

[40]  Katsumi Inoue,et al.  Generating Combinatorial Test Cases by Efficient SAT Encodings Suitable for CDCL SAT Solvers , 2010, LPAR.

[41]  J. Czerwonka.  Pairwise Testing in the Real World : Practical Extensions to Test-Case Scenarios , 2011 .

[42]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[43]  Kaile Su,et al.  Configuration Checking with Aspiration in Local Search for SAT , 2012, AAAI.

[44]  Kaile Su,et al.  Improving Local Search for Random 3-SAT Using Quantitative Configuration Checking , 2012, ECAI.

[45]  Kaile Su,et al.  Local search for Boolean Satisfiability with configuration checking and subscore , 2013, Artif. Intell..

[46]  Myra B. Cohen,et al.  Efficiency and early fault detection with lower and higher strength combinatorial interaction testing , 2013, ESEC/FSE 2013.

[47]  Jeff Yu Lei,et al.  An Efficient Algorithm for Constraint Handling in Combinatorial Test Generation , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[48]  Yu Lei,et al.  Introduction to Combinatorial Testing , 2013 .

[49]  Jian Zhang,et al.  Automatic Generation of Combinatorial Test Data , 2014, SpringerBriefs in Computer Science.

[50]  Jun Yan,et al.  Generating combinatorial test suite using combinatorial optimization , 2014, J. Syst. Softw..

[51]  Kaile Su,et al.  More efficient two-mode stochastic local search for random 3-satisfiability , 2014, Applied Intelligence.

[52]  Kaile Su,et al.  Tailoring Local Search for Partial MaxSAT , 2014, AAAI.

[53]  Wei Wu,et al.  Double Configuration Checking in Stochastic Local Search for Satisfiability , 2014, AAAI.

[54]  Wei Wu,et al.  Clause States Based Configuration Checking in Local Search for Satisfiability , 2015, IEEE Transactions on Cybernetics.

[55]  Wei Wu,et al.  CCLS: An Efficient Local Search Algorithm for Weighted Maximum Satisfiability , 2015, IEEE Transactions on Computers.

[56]  Myra B. Cohen,et al.  Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[57]  Armin Biere,et al.  Optimization of Combinatorial Testing by Incremental SAT Solving , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).