Decreasing the cost of mutation testing with second-order mutants

Although powerful, mutation is a computationally very expensive testing technique. In fact, its three main stages (mutant generation, mutant execution and result analysis) require many resources to be successfully accomplished. Thus, researchers have made important efforts to reduce its costs. This paper represents an additional effort in this sense. It describes the results of two experiments in which, by means of combining the original set of mutants and therefore obtaining a new set of mutants—each one with two faults—the number of mutants used is reduced to half. Results lead to believe that mutant combination does not decrease the quality of the test suite, whereas it supposes important savings in mutant execution and result analysis. Copyright © 2008 John Wiley & Sons, Ltd.

[1]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[2]  A.P. Mathur Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[3]  Ishbel M. M. Duncan Strong mutation testing strategies , 1993 .

[4]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[5]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[6]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[7]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[8]  Vladimir N. Fleyshgakker,et al.  Improved serial algorithms for mutation analysis , 1993, ISSTA '93.

[9]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[10]  John A. Clark,et al.  Investigating the effectiveness of object‐oriented testing strategies using the mutation method , 2001 .

[11]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[12]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999 .

[13]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[14]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999 .

[15]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[16]  Mario Piattini,et al.  Integrating techniques and tools for testing automation: Research Articles , 2007 .

[17]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[18]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[19]  A. Jefferson Offutt,et al.  Inter-class mutation operators for Java , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[20]  Aditya P. Mathur,et al.  PMothra: scheduling mutants for execution on a hypercube , 1989 .

[21]  Osamu Mizuno,et al.  Elimination of crucial faults by a new selective testing method , 2002, Proceedings International Symposium on Empirical Software Engineering.

[22]  Mario Piattini,et al.  Integrating techniques and tools for testing automation , 2007, Softw. Test. Verification Reliab..

[23]  Richard A. DeMillo,et al.  Compiler-integrated program mutation , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[24]  Byoungju Choi,et al.  High-performance mutation testing , 1993, J. Syst. Softw..

[25]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[26]  Y. L. Traon,et al.  From genetic to bacteriological algorithms for mutation‐based testing , 2005, Softw. Test. Verification Reliab..

[27]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[28]  Vernon Rego,et al.  High Performance Software Testing on SIMD Machines , 1991, IEEE Trans. Software Eng..

[29]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.