Improving Combinatorial Methods to Optimise Efficient Test Suite Generation

Complex software systems typically contain many components that often take a broad range of input variables. Exhaustive system testing will reveal defects by exercising all combinations of components and variables. Since exhaustive testing of this sort often takes too long to plan and execute, more efficient methods are required that optimise requirement and code coverage against test effort. Pair-wise testing is one of many 'black-box' techniques that can help solve this problem, but it is often necessary to add tests beyond pair-wise to increase coverage while balancing the test generation efficiency. We have created a configurable tool to generate not only pair-wise, but also 3-wise, 4-wise n-wise test suites to increase coverage. The tool iterates through component combination thresholds to determine the suite with the fewest test cases for each test scenario. This experience report will present our findings when the tool was used to test a set of features in a real world release of a data integration application.