Are We There Yet? How Redundant and Equivalent Mutants Affect Determination of Test Completeness

Mutation score has long been used in research as a metric to measure the effectiveness of testing strategies. This paper presents evidence that mutation score lacks the desired accuracy to determine the completeness of a test suite due to noise introduced by the redundancy inherent in traditional mutation, and that dominator mutation score is a superior metric for this purpose. We evaluate the impact of different levels of redundant and equivalent mutants on mutation score and the ability to determine completeness in developing a mutation-adequate test suite. We conclude that, in the context of our model, redundant mutants make it very difficult to accurately assess test completeness. Equivalent mutants, on the other hand, have little effect on determining completeness. Based on this information, we suggest limits to redundancy and equivalency that mutation tools must achieve to be practical for general use in software testing.

[1]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[2]  René Just,et al.  Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[3]  Akbar Siami Namin,et al.  Sufficient mutation operators for measuring test effectiveness , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[4]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[5]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[6]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[7]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[8]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[9]  A. Jefferson Offutt,et al.  Experimental Evaluation of SDL and One-Op Mutation for C , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[10]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[11]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[12]  René Just,et al.  MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[13]  A. Jefferson Offutt,et al.  Better predicate testing , 2011, AST '11.

[14]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[15]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[16]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[17]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[18]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[19]  Mark Harman,et al.  A study of equivalent and stubborn mutation operators using human analysis of equivalence , 2014, ICSE.

[20]  J.H. Andrews,et al.  Finding Sufficient Mutation Operators via Variable Reduction , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[21]  A. Jefferson Offutt,et al.  Improving logic-based testing , 2013, J. Syst. Softw..

[22]  A. Jefferson Offutt,et al.  Growing a Reduced Set of Mutation Operators , 2014, 2014 Brazilian Symposium on Software Engineering.

[23]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[24]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997, Softw. Test. Verification Reliab..

[25]  A.P. Mathur.  Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[26]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[27]  René Just,et al.  Higher accuracy and lower run time: efficient mutation analysis using non‐redundant mutation operators , 2015, Softw. Test. Verification Reliab..

[28]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[29]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[30]  A. Jefferson Offutt,et al.  Mutant Subsumption Graphs , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[31]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[32]  A. Jefferson Offutt,et al.  Empirical Evaluation of the Statement Deletion Mutation Operator , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[33]  MaYu-Seung,et al.  MuJava: an automated class mutation system , 2005 .

[34]  Akbar Siami Namin,et al.  On Sufficiency of Mutants , 2007, 29th International Conference on Software Engineering (ICSE'07 Companion).

[35]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[36]  Alex Groce,et al.  How hard does mutation analysis have to be, anyway? , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[37]  Mario Jino,et al.  Proteum: uma ferramenta de teste baseada na analise de mutantes , 1993 .

[38]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..