Avoiding useless mutants

Mutation testing is a program-transformation technique that injects artificial bugs to check whether the existing test suite can detect them. However, the costs of using mutation testing are usually high, hindering its use in industry. Useless mutants (equivalent and duplicated) contribute to increase costs. Previous research has focused mainly on detecting useless mutants only after they are generated and compiled. In this paper, we introduce a strategy to help developers with deriving rules to avoid the generation of useless mutants. To use our strategy, we pass as input a set of programs. For each program, we also need a passing test suite and a set of mutants. As output, our strategy yields a set of useless mutants candidates. After manually confirming that the mutants classified by our strategy as "useless" are indeed useless, we derive rules that can avoid their generation and thus decrease costs. To the best of our knowledge, we introduce 37 new rules that can avoid useless mutants right before their generation. We then implement a subset of these rules in the MUJAVA mutation testing tool. Since our rules have been derived based on artificial and small Java programs, we take our MUJAVA version embedded with our rules and execute it in industrial-scale projects. Our rules reduced the number of mutants by almost 13% on average. Our results are promising because (i) we avoid useless mutants generation; (ii) our strategy can help with identifying more rules in case we set it to use more complex Java programs; and (iii) our MUJAVA version has only a subset of the rules we derived.

[1]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[2]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[3]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[4]  Gustavo Soares,et al.  Automated Behavioral Testing of Refactoring Engines , 2013, IEEE Transactions on Software Engineering.

[5]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[6]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[7]  Mark Harman,et al.  Detecting Trivial Mutant Equivalences via Compiler Optimisations , 2018, IEEE Transactions on Software Engineering.

[8]  Mike Papadakis,et al.  Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study , 2016, 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[9]  Andreas Zeller,et al.  Covering and Uncovering Equivalent Mutants , 2013, Softw. Test. Verification Reliab..

[10]  Rohit Gheyi,et al.  Making Program Refactoring Safer , 2010, IEEE Software.

[11]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[12]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[13]  Cyrille Artho,et al.  Classification of Randomly Generated Test Cases , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[14]  Gary McGraw,et al.  Software fault injection: inoculating programs against errors , 1997 .

[15]  Nicos Malevris,et al.  MEDIC: A static analysis framework for equivalent mutant identification , 2015, Inf. Softw. Technol..

[16]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[17]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997, Softw. Test. Verification Reliab..

[18]  Daniel Jackson,et al.  Software Abstractions - Logic, Language, and Analysis , 2006 .

[19]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[20]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[21]  Daniel Jackson,et al.  Alcoa: the Alloy constraint analyzer , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[22]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[23]  René Just,et al.  MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[24]  A. Jefferson Offutt,et al.  The class-level mutants of MuJava , 2006, AST '06.

[25]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[26]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[27]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[28]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[29]  René Just,et al.  Inferring mutant utility from program context , 2017, ISSTA.

[30]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[31]  Melina Mongiovi,et al.  Scaling Testing of Refactoring Engines , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[32]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[33]  J. A. Acree On mutation , 1980 .

[34]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[35]  Corina S. Pasareanu,et al.  Symbolic PathFinder: symbolic execution of Java bytecode , 2010, ASE.