A backtracking search tool for constructing combinatorial test suites

Combinatorial testing is an important testing method. It requires the test cases to cover various combinations of parameters of the system under test. The test generation problem for combinatorial testing can be modeled as constructing a matrix which has certain properties. This paper first discusses two combinatorial testing criteria: covering array and orthogonal array, and then proposes a backtracking search algorithm to construct matrices satisfying them. Several search heuristics and symmetry breaking techniques are used to reduce the search time. This paper also introduces some techniques to generate large covering array instances from smaller ones. All the techniques have been implemented in a tool called EXACT (EXhaustive seArch of Combinatorial Test suites). A new optimal covering array is found by this tool.

[1]  Patrick J. Schroeder,et al.  Comparing the fault detection effectiveness of n-way and random test suites , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[2]  Gadiel Seroussi,et al.  Vector sets for exhaustive testing of logic circuits , 1988, IEEE Trans. Inf. Theory.

[3]  Siddhartha R. Dalal,et al.  Model-based testing in practice , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[4]  Charles J. Colbourn,et al.  Covering Arrays of Strength Three , 1999, Des. Codes Cryptogr..

[5]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[6]  Toby Walsh,et al.  Breaking Row and Column Symmetries in Matrix Models , 2002, CP.

[7]  Donald L. Kreher,et al.  On the state of strength‐three covering arrays , 2002 .

[8]  C. Colbourn.  Combinatorial aspects of covering arrays , 2006 .

[9]  Yu-Wen Tung,et al.  Automating test case generation for the new generation mission software system , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[10]  Christine T. Cheng,et al.  Generating small combinatorial test suites to cover input-output relationships , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[11]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[12]  Andries E. Brouwer,et al.  Orthogonal arrays of strength 3 and small run sizes , 2006 .

[13]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[14]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[15]  Jun Yan,et al.  Backtracking Algorithms and Search Heuristics to Generate Test Suites for Combinatorial Testing , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[16]  H. Kopka,et al.  Guide to LaTeX , 1999 .

[17]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[18]  Robert L. Probert,et al.  A practical strategy for testing pair-wise coverage of network interfaces , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[19]  Steven David Prestwich,et al.  Constraint Models for the Covering Test Problem , 2006, Constraints.

[20]  Tran van Trung,et al.  On t-Covering Arrays , 2004, Des. Codes Cryptogr..

[21]  Brett Stevens,et al.  Covering arrays on graphs: Qualitative independence graphs and extremal set partition theory , 2007, math/0701553.

[22]  Kenneth H. Rosen,et al.  Discrete Mathematics and its applications , 2000 .

[23]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[24]  Myra B. Cohen,et al.  Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces , 2006, IEEE Trans. Software Eng..

[25]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[26]  Anant P. Godbole,et al.  t-Covering Arrays: Upper Bounds and Poisson Approximations , 1996, Combinatorics, Probability and Computing.

[27]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[28]  Dennis Shasha,et al.  Products of mixed covering arrays of strength two , 2006 .

[29]  D. Richard Kuhn.  An algorithm for generating very large covering arrays , 2006 .

[30]  Jian Zhang.  Automatic symmetry breaking method combined with SAT , 2001, SAC.

[31]  E. Mendelsohn,et al.  New recursive methods for transversal covers , 1999 .