Efficient Mutation Testing using Whole Test Suite Generation

By seeding artificial faults (mutants), mutation testing can tell us how good existing tests are, and it can help to direct test generation efforts. However, mutation based test generation is hampered because there usually are simply too many mutants, and too many of these mutants are either trivially killed or equivalent. Any effort spent on test generation for equivalent mutants is per definition wasted, and misdirects resources from where they could be put to better use – for example to produce tests revealing more non-equivalent mutants. To overcome this problem, our search-based EVOSUITE test generation tool integrates two optimizations: First, we avoid redundant test executions on mutants by monitoring state infection conditions, and second we use whole test suite generation to optimize test suites towards killing the highest number of mutants, rather than selecting individual mutants. These optimizations allowed us to perform one of the largest empirical studies on mutation testing to date, where we applied EVOSUITE to a random sample of 100 open source projects consisting of a total of 8,963 classes, leading to a total of 1,380,302 mutants. The experiment not only demonstrates that our approach scales well, but it also quantifies the relations between weak and strong mutation testing as well as branch coverage, and points out current limitations. Keywords-mutation testing; test case generation; searchbased testing; testing classes; unit testing