On generating mutants for AspectJ programs

Context: Mutation analysis has been widely used in research studies to evaluate the effectiveness of test suites and testing techniques. Faulty versions (i.e., mutants) of a program are generated such that each mutant contains one seeded fault. The mutation score provides a measure of effectiveness. Objective: We study three problems with the use of mutation analysis for testing AspectJ programs:*The manual identification and removal of equivalent mutants is difficult and time consuming. We calculate the percentage of equivalent mutants generated for benchmark AspectJ programs using available mutation tools. *The generated mutants need to cover the various fault types described in the literature on fault models for AspectJ programs. We measure the distribution of the mutants generated using available mutation tools with respect to the AspectJ fault types. *We measure the difficulty of killing the generated mutants. We propose the use of simple analysis of the subject programs to prevent the generation of some equivalent mutants. Method: We revised existing AspectJ fault models and presented a fault model that removes the problems in existing fault models, such as overlapping between fault types and missing fault types. We also defined three new fault types that occur due to incorrect data-flow interactions occurring in AspectJ programs. We used three mutation tools: AjMutator, Proteum/AJ, and MuJava on three AspectJ programs. To measure the difficulty of killing the mutants created using a mutation operator, we compared the average number of the mutants killed by 10 test suites that satisfy block coverage criterion. Results: A high percentage of the mutants are equivalent. The mutation tools do not cover all the fault types. Only 4 out of 27 operators generated mutants that were easy to kill. Conclusions: Our analysis approach removed about 80% of the equivalent mutants. Higher order mutation is needed to cover all the fault types.