catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

We develop an approach to test suite generation for Constrained Combinatorial Testing (CCT), one of the most widely studied combinatorial testing techniques, based on Answer Set Programming (ASP). The resulting catnap system accepts a CCT instance in fact format and combines it with a first-order encoding for generating test suites, which can subsequently be solved by any off-the-shelf ASP systems. We evaluate the effectiveness of our approach by empirically contrasting it to the best known bounds obtained via dedicated implementations.

[1]  Katsumi Inoue,et al.  Generating Event-Sequence Test Cases by Answer Set Programming with the Incidence Matrix , 2012, ICLP.

[2]  J. Czerwonka.  Pairwise Testing in Real World Practical Extensions to Test Case Generators , 2006 .

[3]  C. Colbourn,et al.  Handbook of Combinatorial Designs , 2006 .

[4]  Tatsuhiro Tsuchiya,et al.  Constructing Test Sets for Pairwise Testing: A SAT-Based Approach , 2011, 2011 Second International Conference on Networking and Computing.

[5]  Marcello Balduccini,et al.  Representing Constraint Satisfaction Problems in Answer Set Programming , 2022 .

[6]  Jeff Yu Lei,et al.  An Efficient Algorithm for Constraint Handling in Combinatorial Test Generation , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[7]  Myra B. Cohen,et al.  Efficiency and early fault detection with lower and higher strength combinatorial interaction testing , 2013, ESEC/FSE 2013.

[8]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[9]  Jun Yan,et al.  Generating combinatorial test suite using combinatorial optimization , 2014, J. Syst. Softw..

[10]  Jacek Czerwonka,et al.  Pairwise Testing in Real World , 2006 .

[11]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[12]  Kaile Su,et al.  TCA: An Efficient Two-Mode Meta-Heuristic Algorithm for Combinatorial Test Generation (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[13]  Myra B. Cohen,et al.  An Improved Meta-heuristic Search for Constrained Interaction Testing , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[14]  Martin Gebser,et al.  Constraint Answer Set Solving , 2009, ICLP.

[15]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[16]  Armin Biere,et al.  Optimization of Combinatorial Testing by Incremental SAT Solving , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[17]  Katsumi Inoue,et al.  Answer-Set Programming as a new Approach to Event-Sequence Testing , 2011 .

[18]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[19]  Alex M. Andrew,et al.  Knowledge Representation, Reasoning and Declarative Problem Solving , 2004 .

[20]  Katsumi Inoue,et al.  $${\varvec{teaspoon}}$$teaspoon: solving the curriculum-based course timetabling problems with answer set programming , 2019, Ann. Oper. Res..

[21]  Katsumi Inoue,et al.  Event-sequence testing using answer-set programming , 2012 .

[22]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..