The Impact of Equivalent Mutants

If a mutation is not killed by a test suite, this usuallymeans that the test suite is not adequate. However, itmay also be that the mutant keeps the program’s seman-tics unchanged—and thus cannot be detected by any test.We found such equivalent mutants to be surprisingly com-mon: In an experiment on the JAXEN XPATH query engine,8/20 = 40% of all mutations turned out to be equivalent.Worse, checking the equivalency took us 15 minutes for asingle mutation. Equivalent mutants thus make it impossi-ble to automatically assess test suites by means of mutationtesting. To identify equivalent mutants, we are currently investi-gating the impact of a mutation on the execution: the morea mutation alters the execution, the higher the chance of itbeing non-equivalent. First experiments assessing the im-pact on code coverage are promising.

[1]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[2]  Sudheendra Hangal,et al.  Tracking down software bugs using automatic anomaly detection , 2002, ICSE '02.

[3]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[4]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[5]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[6]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[7]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[8]  Matthew B. Dwyer,et al.  Differential symbolic execution , 2008, SIGSOFT '08/FSE-16.

[9]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[10]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[11]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[12]  Andreas Zeller,et al.  Lightweight Defect Localization for Java , 2005, ECOOP.

[13]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[14]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[15]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[16]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[17]  Frank Tip,et al.  Change impact analysis for object-oriented programs , 2001, PASTE '01.

[18]  S. N. Weiss,et al.  All-Uses versus Mutation Testing : An ExperimentalComparison of E ectiveness , 1996 .