MEDIC: A static analysis framework for equivalent mutant identification

Abstract Context: The equivalent mutant problem is a well-known impediment to the adoption of mutation testing in practice. In consequence of its undecidable nature, a complete automated solution is unattainable. To worsen the situation, the manual analysis of the generated mutants of a program under test is prohibitive due to their vast number and the complexity of determining their equivalence. Objective: This paper focuses on the automated identification of equivalent and partially equivalent mutants, i.e. mutants that are equivalent to the original program for a specific subset of paths. To this end, the utilisation of a series of previously proposed data flow patterns is investigated. This study also examines the cross-language nature of these patterns and the killability of the detected partially equivalent mutants. Method: A tool, named MEDIC (Mutants’ Equivalence DIsCovery), incorporating the aforementioned patterns was developed. Its efficiency and effectiveness were evaluated based on a set of manually analysed mutants from real-world programs, written in the Java programming language. Furthermore, MEDIC was employed to test subjects written in the JavaScript programming language. Results: MEDIC managed to detect 56% of the examined equivalent mutants in 125 s, providing strong evidence regarding both its effectiveness and efficiency. Additionally, MEDIC was able to identify equivalent mutants in the JavaScript test subjects, lending colour to the cross-language nature of the implemented patterns. Finally, the identified partially equivalent mutant set consisted largely of killable mutants, 16% of which were stubborn ones. Conclusion: It can be concluded that pattern-based equivalent mutant identification forms a viable approach for combating the equivalent mutant problem. MEDIC automatically detected a considerable number of the manually identified equivalent mutants and was successfully applied to test subjects in all examined programming languages.

[1]  M. Wegman,et al.  Global value numbers and redundant computations , 1988, POPL '88.

[2]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[3]  Franz Wotawa,et al.  EqMutDetect — A tool for equivalent mutant detection in embedded systems , 2012, Proceedings of the 10th International Workshop on Intelligent Solutions in Embedded Systems.

[4]  Nancy G. Leveson,et al.  Software safety , 1982, ACM SIGSOFT Softw. Eng. Notes.

[5]  Mark N. Wegman,et al.  Efficiently computing static single assignment form and the control dependence graph , 1991, TOPL.

[6]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[7]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[8]  Mike Papadakis,et al.  Mutation based test case generation via a path selection strategy , 2012, Inf. Softw. Technol..

[9]  Sarfraz Khurshid,et al.  Operator-based and random mutant selection: Better together , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[10]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[11]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[12]  Mike Papadakis,et al.  Employing second‐order mutation for isolating first‐order equivalent mutants , 2015, Softw. Test. Verification Reliab..

[13]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[14]  Timothy J. Harvey,et al.  Practical improvements to the construction and destruction of static single assignment form , 1998 .

[15]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[16]  Mark Harman,et al.  A study of equivalent and stubborn mutation operators using human analysis of equivalence , 2014, ICSE.

[17]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[18]  Nicos Malevris,et al.  Identifying More Equivalent Mutants via Code Similarity , 2013, 2013 20th Asia-Pacific Software Engineering Conference (APSEC).

[19]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[20]  A. Jefferson Offutt,et al.  The class-level mutants of MuJava , 2006, AST '06.

[21]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[22]  Mike Papadakis,et al.  Isolating First Order Equivalent Mutants via Second Order Mutation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[23]  Pacific Biological Station.  Proceedings of the Second Workshop , 2000 .

[24]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second-order mutants , 2009 .

[25]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[26]  Mike Papadakis,et al.  Evaluating Mutation Testing Alternatives: A Collateral Experiment , 2010, 2010 Asia Pacific Software Engineering Conference.

[27]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[28]  W. Eric Wong,et al.  An empirical comparison of data flow and mutation‐based test adequacy criteria , 1994, Softw. Test. Verification Reliab..

[29]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[30]  Gary McGraw,et al.  Software fault injection: inoculating programs against errors , 1997 .

[31]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[32]  Nicos Malevris,et al.  Using Data Flow Patterns for Equivalent Mutant Detection , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[33]  Fan Wu,et al.  Mutation testing of memory-related operators , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[34]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[35]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[36]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[37]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[38]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[39]  Auri Marcelo Rizzo Vincenzi,et al.  Proteum: a family of tools to support specification and program testing based on mutation , 2001 .

[40]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[41]  Sebastian Buchwald,et al.  Simple and Efficient Construction of Static Single Assignment Form , 2013, CC.

[42]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[43]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[44]  Franz Wotawa,et al.  Using Constraints for Equivalent Mutant Detection , 2012, WS-FMDS.

[45]  Andreas Zeller,et al.  Covering and Uncovering Equivalent Mutants , 2013, Softw. Test. Verification Reliab..

[46]  Jong-Deok Choi,et al.  Automatic construction of sparse data flow evaluation graphs , 1991, POPL '91.

[47]  Mark Harman,et al.  The relationship between program dependence and mutation analysis , 2001 .

[48]  A. Jefferson Offutt,et al.  Experiments with Data Flow and Mutation Testing , 1994 .

[49]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[50]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[51]  Yong Rae Kwon,et al.  MuJava: an automated class mutation system: Research Articles , 2005 .

[52]  Bowen Alpern,et al.  Detecting equality of variables in programs , 1988, POPL '88.

[53]  A. Jefferson Offutt,et al.  An Experimental Evaluation of Data Flow and Mutation Testing , 1996 .

[54]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2012, IEEE Trans. Software Eng..

[55]  R.A. DeMillo,et al.  An extended overview of the Mothra software testing environment , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[56]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.