MutantBench: an Equivalent Mutant Problem Comparison Framework

The Equivalent Mutant Problem is an ongoing research problem that led to the creation of multiple reusable non-standardized mutant datasets. However, cross-study evaluation is still tedious. To tackle this problem, we propose MutantBench, a novel open-source comparison framework that is designed with a focus on the FAIR data principles and adoptability by the community. With this, tools within the Equivalent Mutant Problem can be empirically evaluated using the same dataset which then allows for improved cross-study evaluation. We also combine existing datasets resulting in a mutant dataset containing 4400 mutants with 1416 equivalent mutants. This increases the previously largest mutant dataset by more than a thousand equivalent mutants.

[1]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[2]  Jóakim von Kistowski,et al.  How to Build a Benchmark , 2015, ICPE.

[3]  Erik Schultes,et al.  The FAIR Guiding Principles for scientific data management and stewardship , 2016, Scientific Data.

[4]  Yves Le Traon,et al.  Sound and Quasi-Complete Detection of Infeasible Test Requirements , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[5]  Norbert Oster,et al.  MutantDistiller: Using Symbolic Execution for Automatic Detection of Equivalent Mutants and Generation of Mutant Killing Tests , 2020, 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[6]  Marinos Kintis,et al.  Effective methods to tackle the equivalent mutant problem when testing software with mutation , 2016 .

[7]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[8]  René Just,et al.  The major mutation framework: efficient and scalable mutation analysis for Java , 2014, ISSTA 2014.

[9]  Matias Martinez,et al.  Fine-grained and accurate source code differencing , 2014, ASE.

[10]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[11]  Mark Harman,et al.  Detecting Trivial Mutant Equivalences via Compiler Optimisations , 2018, IEEE Transactions on Software Engineering.

[12]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[13]  Mark Harman,et al.  A study of equivalent and stubborn mutation operators using human analysis of equivalence , 2014, ICSE.

[14]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[15]  Jianbin Fang,et al.  A Comprehensive Performance Comparison of CUDA and OpenCL , 2011, 2011 International Conference on Parallel Processing.

[16]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[17]  G. Fitzgerald,et al.  'I. , 2019, Australian journal of primary health.

[18]  Mahdi Houshmand,et al.  TCE+: An Extension of the TCE Method for Detecting Equivalent Mutants in Java Programs , 2017, FSEN.

[19]  Yves Le Traon,et al.  Muteria: An Extensible and Flexible Multi-Criteria Software Testing Framework , 2020, AST@ICSE.

[20]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[21]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[22]  Susan Elliott Sim,et al.  Using benchmarking to advance research: a challenge to software engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[23]  Mike Papadakis,et al.  Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study , 2016, 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM).