Assessing the Applicability of a Combinatorial Testing tool within an Industrial Environment

This paper describes a case study executed to evaluate a combinatorial test design approach within the industrial setting of IBM Research. An existing combinatorial test suite was compared against a prioritized combinatorial test suite that was generated with the Combinatorial Tree Editor XL Profesional (CTE). The prioritization technique was recently developed and added to the CTE in the context of the FITTEST project. Test design for the new test suite was carried out by the developers of the prioritization technique. Test implementation and execution was done by the industrial partner of the System Under Test. This case study has investigated whether the prioritized combinatorial technique is useful to complement current testing practices at IBM Research. The focus of this study is on fault finding capability of artificially injected faults that have been selected and prioritized using domain knowledge and expertise, and efficiency of test case execution. Conclusions of this study are that for the testing of the target product in a simulated environment, the improved combinatorial testing tools do qualify as useful and this type of testing will be included in current practices.