Empirical Evaluation of the Statement Deletion Mutation Operator

Mutation analysis is widely considered to be an exceptionally effective criterion for designing tests. It is also widely considered to be expensive in terms of the number of test requirements and in the amount of execution needed to create a good test suite. This paper posits that simply deleting statements, implemented with the statement deletion (SDL) mutation operators in Mothra, is enough to get very good tests. A version of the SDL operator for Java was designed and implemented inside the muJava mutation system. The SDL operator was applied to 40 separate Java classes, tests were designed to kill the non-equivalent SDL mutants, and then run against all mutants.

[1]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[2]  R. Lipton,et al.  Mutation analysis , 1998 .

[3]  Márcio Eduardo Delamaro,et al.  Proteum - A Tool for the Assessment of Test Adequacy for C Programs User's guide , 1996 .

[4]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[5]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[6]  Weichen Eric Wong.  On mutation and data flow , 1993 .

[7]  Akbar Siami Namin,et al.  Sufficient mutation operators for measuring test effectiveness , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[8]  Aditya P. Mathur,et al.  High performance testing on SIMD machines , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[9]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[10]  Richard A. DeMillo,et al.  Compiler-integrated program mutation , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[11]  A. Jefferson Offutt,et al.  An Experimental Evaluation of Data Flow and Mutation Testing , 1996 .

[12]  Yves Crouzet,et al.  An experimental study on software structural testing: deterministic versus random input generation , 1991, [1991] Digest of Papers. Fault-Tolerant Computing: The Twenty-First International Symposium.

[13]  John A. Clark,et al.  Investigating the effectiveness of object‐oriented testing strategies using the mutation method , 2001 .

[14]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[15]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[16]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[17]  Roy P. Pargas,et al.  Mutation Testing of Software Using MIMD Computer , 1992, ICPP.

[18]  Byoungju Choi,et al.  High-performance mutation testing , 1993, J. Syst. Softw..

[19]  Roland H. Untch,et al.  Schema-based mutation analysis: a new test data adequacy assessment method , 1995 .

[20]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[21]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[22]  Phyllis G. Frankl,et al.  Comparison of delivered reliability of branch, data flow and operational testing: A case study , 2000, ISSTA '00.

[23]  Aditya P. Mathur,et al.  Modeling mutation and a vector processor , 1988, Proceedings. [1989] 11th International Conference on Software Engineering.

[24]  Aditya P. Mathur,et al.  Modeling mutation on a vector processor , 1988, ICSE '88.

[25]  A. Jefferson Offutt,et al.  Mutation Operators for Ada , 1996 .

[26]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[27]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[28]  Roland H. Untch.  On reduced neighborhood mutation analysis using a single mutagenic operator , 2009, ACM-SE 47.

[29]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[30]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[31]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[32]  Eugene H. Spafford,et al.  Critical slicing for software fault localization , 1996, ISSTA '96.

[33]  John A. Clark,et al.  Investigating the effectiveness of object-oriented strategies with the mutation method (abstract only) , 2001 .

[34]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[35]  René Just,et al.  Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[36]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.