Equivalent Mutant Problem and its Problem-Solving Techniques: A Retrospective View

Mutation testing is a fault simulating testing technique mainly used to assess the quality of the test suite. During the execution of mutants, some of the mutants remain undetected are known to be equivalent. They also make the mutation testing process costly and time-consuming. Due to the similar behavior of the original program and equivalent mutants, they are unable to be identified by any of the test suites. Researchers have been working in the field to identify and detect equivalent mutants manually and automatically. In this paper, a comprehensive study is done to analyze the problem of equivalent mutants. During the course of this review, several well-known databases were searched for the studies discussing the equivalent mutant problem. A total of 36 studies were found to be relevant to the topic and were evaluated. It is concluded that handling equivalent mutants is the most critical problem; it negatively impacts the test adequacy score. The research work can be categorized into three categories: equivalent mutant detection, how to avoid the generation of equivalent mutants and analyzing the impact of equivalent mutants on different parameters of testing. Compiler optimization techniques and higher order mutation techniques are found to be the most prevailing to distinguish the equivalent mutants from a pool of mutants.

[1]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[2]  René Just,et al.  Higher accuracy and lower run time: efficient mutation analysis using non‐redundant mutation operators , 2015, Softw. Test. Verification Reliab..

[3]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[4]  Mike Papadakis,et al.  Employing second‐order mutation for isolating first‐order equivalent mutants , 2015, Softw. Test. Verification Reliab..

[5]  A. Jefferson Offutt,et al.  Analyzing the validity of selective mutation with dominator mutants , 2016, SIGSOFT FSE.

[6]  A. Jefferson Offutt,et al.  The class-level mutants of MuJava , 2006, AST '06.

[7]  Mike Papadakis,et al.  Evaluating Mutation Testing Alternatives: A Collateral Experiment , 2010, 2010 Asia Pacific Software Engineering Conference.

[8]  András Márki,et al.  On strong mutation and subsuming mutants , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[9]  Mark Harman,et al.  A study of equivalent and stubborn mutation operators using human analysis of equivalence , 2014, ICSE.

[10]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[11]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[12]  Franz Wotawa,et al.  EqMutDetect — A tool for equivalent mutant detection in embedded systems , 2012, Proceedings of the 10th International Workshop on Intelligent Solutions in Embedded Systems.

[13]  Ali Mesbah,et al.  Efficient JavaScript Mutation Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[14]  Roseli A. Francelin Romero,et al.  Bayesian-Learning Based Guidelines to Determine Equivalent Mutants , 2001, Int. J. Softw. Eng. Knowl. Eng..

[15]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[16]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[17]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[18]  Nicos Malevris,et al.  Identifying More Equivalent Mutants via Code Similarity , 2013, 2013 20th Asia-Pacific Software Engineering Conference (APSEC).

[19]  Sunil Kumar Khatri,et al.  Experimental comparison of automated mutation testing tools for java , 2015, 2015 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions).

[20]  A. Jefferson Offutt,et al.  Are We There Yet? How Redundant and Equivalent Mutants Affect Determination of Test Completeness , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[21]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[22]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[23]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[24]  Jatana Nishtha,et al.  Systematic Literature Review on Search Based Mutation Testing , 2017 .

[25]  Robert M. Hierons,et al.  Resolving the Equivalent Mutant Problem in the Presence of Non-determinism and Coincidental Correctness , 2016, ICTSS.

[26]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[27]  W. Eric Wong.  Mutation Testing for the New Century , 2001 .

[28]  Yves Le Traon,et al.  Mitigating the effects of equivalent mutants with mutant classification strategies , 2014, Sci. Comput. Program..

[29]  Franz Wotawa,et al.  Using Constraints for Equivalent Mutant Detection , 2012, WS-FMDS.

[30]  Phil McMinn,et al.  The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis , 2014, 2014 14th International Conference on Quality Software.

[31]  Mark Harman,et al.  Detecting Trivial Mutant Equivalences via Compiler Optimisations , 2018, IEEE Transactions on Software Engineering.

[32]  Andreas Zeller,et al.  Covering and Uncovering Equivalent Mutants , 2013, Softw. Test. Verification Reliab..

[33]  Morgan B Kaufmann,et al.  Mutation Testing for the New Century , 2002, Computer.

[34]  Paolo Arcaini,et al.  A novel use of equivalent mutants for static anomaly detection in software artifacts , 2017, Inf. Softw. Technol..

[35]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[36]  Nicos Malevris,et al.  MEDIC: A static analysis framework for equivalent mutant identification , 2015, Inf. Softw. Technol..

[37]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[38]  Ali Mesbah,et al.  Guided Mutation Testing for JavaScript Web Applications , 2015, IEEE Transactions on Software Engineering.

[39]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[40]  Nicos Malevris,et al.  Using Data Flow Patterns for Equivalent Mutant Detection , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[41]  Mike Papadakis,et al.  Isolating First Order Equivalent Mutants via Second Order Mutation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[42]  Márcio Eduardo Delamaro,et al.  Proteum - A Tool for the Assessment of Test Adequacy for C Programs User's guide , 1996 .

[43]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[44]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[45]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[46]  Auri Marcelo Rizzo Vincenzi,et al.  Muta-Pro: Towards the definition of a mutation testing process , 2006, Journal of the Brazilian Computer Society.