Reducing Combinatorial Testing Requirements Based on Equivalences with Respect to the Code Under Test

Combinatorial testing (also known as “N-wise testing” or “combinatorial interaction testing”) is a wellstudied approach for black-box testing of software systems [Grindal et al. 2005; Ammann and Offutt 2008; Nie and Leung 2011]. For example, two surveys of combinatorial testing [Grindal et al. 2005; Nie and Leung 2011] cover dozens of techniques and are themselves cited over 430 times each (according to Google Scholar on July 15, 2018). Combinatorial testing requires the user to provide domains of values for each parameter input of the software system under test, and then creates test inputs by different combinations of parameter values. Researchers have defined several coverage criteria for combinatorial testing. The criteria range from the weakest base-choice, to stronger and widely used pairwise criterion (also known as “all-pairs criterion”), to even stronger N-wise (for N > 2), up to the most comprehensive criterion all combinations coverage. As the criterion name implies, all combinations coverage requires the entire cross-product of all parameter values from the input space to be used for testing. While using this criterion gives the most assurance in the correctness of the code under test, this criterion can have too many test requirements, which can make it impractical to apply. For example, if there are p parameters, each with only 2 values, then the total number of combinations is 2. This paper introduces a novel, simple approach that provides the same assurance as all combinations coverage but typically with fewer test inputs, thereby reducing the overall cost of combinatorial testing.