Using Evolutionary Mutation Testing to improve the quality of test suites

Mutation testing is a method used to assess and improve the fault detection capability of a test suite by creating faulty versions, called mutants, of the system under test. Evolutionary Mutation Testing (EMT), like selective mutation or mutant sampling, was proposed to reduce the computational cost, which is a major concern when applying mutation testing. This technique implements an evolutionary algorithm to produce a reduced subset of mutants but with a high proportion of mutants that can help the tester derive new test cases (strong mutants). In this paper, we go a step further in estimating the ability of this technique to induce the generation of test cases. Instead of measuring the percentage of strong mutants within the subset of generated mutants, we compute how much the test suite is actually improved thanks to those mutants. In our experiments, we have compared the extent to which EMT and the random selection of mutants help to find missing test cases in C++ object-oriented systems. We can conclude from our results that the percentage of mutants generated with EMT is lower than with the random strategy to obtain a test suite of the same size and that the technique scales better for complex programs.

[1]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[2]  John J. Marciniak.  Encyclopedia of software engineering (vol. 1 A-N) , 1994 .

[3]  Inmaculada Medina-Bulo,et al.  GiGAn: evolutionary mutation testing for C++ object-oriented systems , 2017, SAC.

[4]  Aurora Trinidad Ramirez Pozo,et al.  Selecting mutation operators with a multiobjective approach , 2012, Expert Syst. Appl..

[5]  Simone do Rocio Senger de Souza,et al.  A systematic review on search based mutation testing , 2017, Inf. Softw. Technol..

[6]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[7]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[8]  Inmaculada Medina-Bulo,et al.  Analogies and Differences between Mutation Operators for WS-BPEL 2.0 and Other Languages , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[9]  Inmaculada Medina-Bulo,et al.  GAmera: An Automatic Mutant Generation System for WS-BPEL Compositions , 2009, 2009 Seventh IEEE European Conference on Web Services.

[10]  L. Darrell Whitley,et al.  HOMAJ: A Tool for Higher Order Mutation Testing in AspectJ and Java , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[11]  Baowen Xu,et al.  Application of Genetic Algorithms in Software Testing , 2007 .

[12]  Auri Marcelo Rizzo Vincenzi,et al.  A coevolutionary algorithm to automatic test case selection and mutant in Mutation Testing , 2013, 2013 IEEE Congress on Evolutionary Computation.

[13]  Aamer Nadeem,et al.  A fitness function for evolutionary mutation testing of object-oriented programs , 2013, 2013 IEEE 9th International Conference on Emerging Technologies (ICET).

[14]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[15]  Inmaculada Medina-Bulo,et al.  Quality metrics for mutation testing with applications to WS‐BPEL compositions , 2015, Softw. Test. Verification Reliab..

[16]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[17]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999 .

[19]  Andreas Zeller,et al.  Breeding High-Impact Mutations , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[20]  Contributor Jennifer A. W. Wright,et al.  Encyclopedia of Information Science and Technology, Third Edition , 2014 .

[21]  Inmaculada Medina-Bulo,et al.  Evolutionary mutation testing , 2011, Inf. Softw. Technol..

[22]  Inmaculada Medina-Bulo,et al.  Class mutation operators for C++ object-oriented systems , 2015, Ann. des Télécommunications.

[23]  Charles Møller,et al.  Encyclopedia of Information Science and Technology , 2005 .

[24]  Inmaculada Medina-Bulo,et al.  Assessment of class mutation operators for C++ with the MuCPP mutation system , 2017, Inf. Softw. Technol..

[25]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[26]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[27]  W. Marsden.  I and J , 2012 .

[28]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[29]  Macario Polo,et al.  Mutation Testing Cost Reduction Techniques: A Survey , 2010, IEEE Software.

[30]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.