Definition and evaluation of mutation operators for GUI-level mutation analysis

Automated testing has become essential in software industry to meet market demands for faster delivery and higher quality software. Testing is performed on many levels of system abstraction, from tests on source code to Graphical User Interface (GUI) tests. New testing techniques and frameworks are also continuously released to the market. Mutation analysis has been proposed as a way of assessing the quality of these new test techniques/frameworks as well as existing test suites in practice. The analysis is performed by seeding defects, referred to as mutants, into the system under test with the assumption that a technique/test suite of high quality will “kill” the mutants. However, whilst support for mutation analysis exists for test techniques that operate on on lower levels of system abstraction, i.e. method-level mutation operators, the support for GUI-level mutation analysis is currently lacking. In this paper we perform an empirical analysis of 18 GUI-level mutation operators defined in our previous work and compare their efficiency and comprehensiveness to state-of-practice lower level mutation operators. The main findings of our analysis are (1) that traditional method-level mutation operators are not precise enough for GUI-level mutation; (2) the defined GUI-based mutation operators provide comprehensive support for GUI-level mutation; and (3) GUI-based mutation operators can be automated but are challenged by the dependencies between GUI widgets.