Towards the assessment of software product line tests: a mutation system for variable systems

Software product line engineering is an emerging methodology for the development of variant-rich software systems. As software product lines are viable for this purpose, testing them is complicated in contrast to non-variable systems, as there is an increasingly amount of possible products due to the number of features. There exist many methods proposed for testing software product lines, but seldom the quality of the resulting tests was assessed. For assessing test quality mutation analysis is a well-known technique and is frequently applied to non-variable software systems. However, mutation analysis cannot be applied straight-forward onto software product lines. We present a mutation system for assessing the quality of software product line tests by means of fault detection capability. Our mutation system comprises model-based mutation operators, test case adaption, automated model and test execution, and automated mutation analysis. So far, we developed several mutation operators for feature models, UML state machines, and mapping models. We evaluated the mutation operators against tests that were generated from the specifications and applied them for three case studies. From the results we draw conclusions about the effectiveness of the individual mutation operators.