On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs

Combinatorial test suite design is a test generation technique, popular in part due to its ability to achieve coverage and defect finding power approximating that of exhaustive testing while keeping test suite sizes constrained. In recent years, there have been numerous advances in combinatorial test design techniques, in terms of efficiency and usability of methods used to create them as well as in understanding of their benefits and limitations when applied to real world software. Numerous case studies have appeared presenting practical applications of the combinatorial test suite design techniques, often comparing them with manually-created, random, or exhaustive suites. These comparisons are done either in terms of defects found or by applying some code coverage metric. Since many different and valid combinatorial test suites of strength t can be created for a given test domain, the question whether they all have the same coverage properties is a pertinent one. In this paper we explore the stability of size and coverage of combinatorial test suites. We find that in general coverage levels increase and coverage variability decreases with increasing order of combinations t; however we also find exceptions with implications for practitioners. In addition, we explore cases where coverage achieved by combinatorial test suites of order t applied to the same program is not different from test suites of order t-1. Lastly, we discuss these findings in context of the ongoing practice of applying code coverage metrics to measure effectiveness of combinatorial test suites.