Faster mutation testing inspired by test prioritization and reduction

Mutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test suite how many mutants it kills. A test t is said to kill a mutant m of program p if the output of t on m is different from the output of t on p. The effectiveness of mutation testing in determining the quality of a test suite relies on the ability to apply it using a large number of mutants. However, running many tests against many mutants is time consuming. We present a family of techniques to reduce the cost of mutation testing by prioritizing and reducing tests to more quickly determine the sets of killed and non-killed mutants. Experimental results show the effectiveness and efficiency of our techniques.

[1]  Alessandro Orso,et al.  Regression test selection for Java software , 2001, OOPSLA '01.

[2]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[3]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[4]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[5]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[6]  René Just,et al.  Using Non-redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[7]  Tao Xie,et al.  Is operator-based mutant selection superior to random mutant selection? , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[8]  Gilles Pokam,et al.  Selective mutation testing for concurrent code , 2013, ISSTA.

[9]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[10]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[11]  Mark Harman,et al.  Strong higher order mutation-based test data generation , 2011, ESEC/FSE '11.

[12]  Gregg Rothermel,et al.  On-demand test suite reduction , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[13]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[14]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[15]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[16]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[17]  Akbar Siami Namin,et al.  Sufficient mutation operators for measuring test effectiveness , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[18]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[19]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[20]  A.P. Mathur.  Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[21]  Tsong Yueh Chen,et al.  A new heuristic for test suite reduction , 1998, Inf. Softw. Technol..

[22]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[23]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[24]  Alessandro Orso,et al.  Scaling regression testing to large software systems , 2004, SIGSOFT '04/FSE-12.

[25]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[26]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[27]  Sarfraz Khurshid,et al.  An Empirical Study of JUnit Test-Suite Reduction , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[28]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[29]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[30]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[31]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[32]  Gregg Rothermel,et al.  A Static Approach to Prioritizing JUnit Test Cases , 2012, IEEE Transactions on Software Engineering.

[33]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[34]  Gregg Rothermel,et al.  Bridging the gap between the total and additional test-case prioritization strategies , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[35]  Richard A. DeMillo,et al.  Compiler-integrated program mutation , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[36]  Sarfraz Khurshid,et al.  Regression mutation testing , 2012, ISSTA 2012.

[37]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.