A systematic literature review of how mutation testing supports quality assurance processes

Mutation testing has been very actively investigated by researchers since the 1970s, and remarkable advances have been achieved in its concepts, theory, technology, and empirical evidence. While the most influential realisations have been summarised by existing literature reviews, we lack insight into how mutation testing is actually applied. Our goal is to identify and classify the main applications of mutation testing and analyse the level of replicability of empirical studies related to mutation testing. To this aim, this paper provides a systematic literature review on the application perspective of mutation testing based on a collection of 191 papers published between 1981 and 2015. In particular, we analysed in which quality assurance processes mutation testing is used, which mutation tools and which mutation operators are employed. Additionally, we also investigated how the inherent core problems of mutation testing, ie, the equivalent mutant problem and the high computational cost, are addressed during the actual usage. The results show that most studies use mutation testing as an assessment tool targeting unit tests, and many of the supporting techniques for making mutation testing applicable in practice are still underdeveloped. Based on our observations, we made 9 recommendations for future work, including an important suggestion on how to report mutation testing in testing experiments in an appropriate manner.

[1]  Márcio Eduardo Delamaro,et al.  Proteum - A Tool for the Assessment of Test Adequacy for C Programs User's guide , 1996 .

[2]  Darko Marinov,et al.  Comparing and combining test-suite reduction and regression test selection , 2015, ESEC/SIGSOFT FSE.

[3]  Ivan Moore Jester - a JUnit test tester , 2001 .

[4]  Franz Wotawa,et al.  The Right Choice Matters! SMT Solving Substantially Improves Model-Based Debugging of Spreadsheets , 2013, 2013 13th International Conference on Quality Software.

[5]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[6]  Mark Harman,et al.  An analysis of the relationship between conditional entropy and failed error propagation in software testing , 2014, ICSE.

[7]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[8]  Gregory Gay,et al.  Observable modified condition/decision coverage , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[9]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[10]  Yves Le Traon,et al.  Metallaxis‐FL: mutation‐based fault localization , 2015, Softw. Test. Verification Reliab..

[11]  Gregg Rothermel,et al.  The Impact of Concurrent Coverage Metrics on Testing Effectiveness , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[12]  Akbar Siami Namin,et al.  Sufficient mutation operators for measuring test effectiveness , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[13]  Simeon C. Ntafos,et al.  On Required Element Testing , 1984, IEEE Transactions on Software Engineering.

[14]  David Jackson,et al.  Parallel firm mutation of Java programs , 2001 .

[15]  Phil McMinn,et al.  Search-Based Testing of Relational Schema Integrity Constraints Across Multiple Database Management Systems , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[16]  Baowen Xu,et al.  Testing and validating machine learning classifiers by metamorphic testing , 2011, J. Syst. Softw..

[17]  Jean-Marc Jézéquel,et al.  Building trust into OO components using a genetic analogy , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[18]  Mohammad Zulkernine,et al.  MUSIC: Mutation-based SQL Injection Vulnerability Checking , 2008, 2008 The Eighth International Conference on Quality Software.

[19]  Gregory Gay,et al.  The Risks of Coverage-Directed Test Case Generation , 2015, IEEE Transactions on Software Engineering.

[20]  Gregg Rothermel,et al.  Are concurrency coverage metrics effective for testing: a comprehensive empirical investigation , 2015, Softw. Test. Verification Reliab..

[21]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[22]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999 .

[23]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[24]  Byoungju Choi,et al.  Effective test case selection for component customization and its application to Enterprise JavaBeans , 2004, Softw. Test. Verification Reliab..

[25]  Amin Milani Fard,et al.  Leveraging existing tests in automated test generation for web applications , 2014, ASE.

[26]  Vernon Rego,et al.  High Performance Software Testing on SIMD Machines , 1991, IEEE Trans. Software Eng..

[27]  Roland H. Untch,et al.  Mutation-based software testing using program schemata , 1992, ACM Southeast Regional Conference.

[28]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[29]  Anna Derezinska Quality Assessment of Mutation Operators Dedicated for C# Programs , 2006, 2006 Sixth International Conference on Quality Software (QSIC'06).

[30]  Fevzi Belli,et al.  Mutation Testing of "Go-Back" Functions Based on Pushdown Automata , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[31]  Jonas Mockus Introduction to Software , 2000 .

[32]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[33]  L WolfAlexander,et al.  Evaluating Test Suites and Adequacy Criteria Using Simulation-Based Models of Distributed Systems , 2008 .

[34]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[35]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[36]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[37]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[38]  Yves Le Traon,et al.  Using Mutants to Locate "Unknown" Faults , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[39]  Yucheng Zhang,et al.  Assertions are strongly correlated with test suite effectiveness , 2015, ESEC/SIGSOFT FSE.

[40]  Claire Le Goues,et al.  GenProg: A Generic Method for Automatic Software Repair , 2012, IEEE Transactions on Software Engineering.

[41]  Gregg Rothermel,et al.  Can fault‐exposure‐potential estimates improve the fault detection abilities of test suites? , 2002, Softw. Test. Verification Reliab..

[42]  Shin Yoo,et al.  Ask the Mutants: Mutating Faulty Programs for Fault Localization , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[43]  Adam Kiezun,et al.  jFuzz: A Concolic Whitebox Fuzzer for Java , 2009, NASA Formal Methods.

[44]  Baowen Xu,et al.  A Novel Method of Mutation Clustering Based on Domain Analysis , 2009, SEKE.

[45]  Morgan B Kaufmann,et al.  Mutation Testing for the New Century , 2002, Computer.

[46]  W. Eric Wong,et al.  An empirical comparison of data flow and mutation‐based test adequacy criteria , 1994, Softw. Test. Verification Reliab..

[47]  MaYu-Seung,et al.  MuJava: an automated class mutation system , 2005 .

[48]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[49]  Gregory Gay,et al.  Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[50]  Tao Xie,et al.  Applying Interface-Contract Mutation in Regression Testing of Component-Based Software , 2007, 2007 IEEE International Conference on Software Maintenance.

[51]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[52]  Mike Papadakis,et al.  Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study , 2016, 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[53]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[54]  A. Jefferson Offutt,et al.  A mutation carol: Past, present and future , 2011, Inf. Softw. Technol..

[55]  Macario Polo,et al.  Mutation at the multi-class and system levels , 2013, Sci. Comput. Program..

[56]  Mario Piattini,et al.  Mutation Testing , 2014, IEEE Software.

[57]  Kysh Lynn,et al.  Difference between a systematic review and a literature review , 2013 .

[58]  Gregory Gay,et al.  Automated Oracle Data Selection Support , 2015, IEEE Transactions on Software Engineering.

[59]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[60]  Lionel C. Briand,et al.  Adaptive random testing: an illusion of effectiveness? , 2011, ISSTA '11.

[61]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[62]  Gordon Fraser,et al.  Combining Multiple Coverage Criteria in Search-Based Unit Test Generation , 2015, SSBSE.

[63]  J. A. Acree On mutation , 1980 .

[64]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[65]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[66]  S. Inglis,et al.  Jumble Java Byte Code to Measure the Effectiveness of Unit Tests , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[67]  M. R. Woodward Mutation testing-an evolving technique , 1990 .

[68]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[69]  Auri Marcelo Rizzo Vincenzi,et al.  Unit and integration testing strategies for C programs using mutation , 2001, Softw. Test. Verification Reliab..

[70]  M. Ellims,et al.  The Csaw C Mutation Tool: Initial Results , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[71]  Michael R. Lyu,et al.  An empirical study on testing and fault tolerance for software reliability engineering , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[72]  Arie van Deursen,et al.  A Systematic Survey of Program Comprehension through Dynamic Analysis , 2008, IEEE Transactions on Software Engineering.

[73]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[74]  Zude Li,et al.  Diagnosing new faults using mutants and prior faults (NIER track) , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[75]  Shaoying Liu,et al.  Generating test data from SOFL specifications , 1999, J. Syst. Softw..

[76]  Phil McMinn,et al.  The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints , 2015, ACM Trans. Softw. Eng. Methodol..

[77]  Christof Fetzer,et al.  Assertion-Driven Development: Assessing the Quality of Contracts Using Meta-Mutations , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[78]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[79]  Zhenyu Chen,et al.  Similarity-based test case prioritization using ordered sequences of program entities , 2013, Software Quality Journal.

[80]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[81]  James M. Bieman,et al.  Using machine learning techniques to detect metamorphic relations for programs without test oracles , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[82]  Rex Black,et al.  Foundations of Software Testing ISTQB Certification , 2006 .

[83]  Mike Papadakis,et al.  Mutation based test case generation via a path selection strategy , 2012, Inf. Softw. Technol..

[84]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997, Softw. Test. Verification Reliab..

[85]  Akbar Siami Namin,et al.  On Sufficiency of Mutants , 2007, 29th International Conference on Software Engineering (ICSE'07 Companion).

[86]  Tao Xie,et al.  A fault model and mutation testing of access control policies , 2007, WWW '07.

[87]  Gordon Fraser,et al.  Sound empirical evidence in software testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[88]  Alexander L. Wolf,et al.  Evaluating Test Suites and Adequacy Criteria Using Simulation-Based Models of Distributed Systems , 2008, IEEE Transactions on Software Engineering.

[89]  Simeon C. Ntafos,et al.  An evaluation of required element testing strategies , 1984, ICSE '84.

[90]  Franz Wotawa,et al.  Why Does my Spreadsheet Compute Wrong Values? , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[91]  Eric Wong,et al.  Testing for security vulnerabilities in software , 2012, ICSE 2012.

[92]  Lu Zhang,et al.  Mutation-based test-case prioritization in software evolution , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[93]  Gregg Rothermel,et al.  A controlled experiment assessing test case prioritization techniques via mutation faults , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[94]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[95]  Jinhui Shan,et al.  An Approach to Test Data Generation for Killing Multiple Mutants , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[96]  Gregg Rothermel,et al.  Test Case Prioritization Based on Information Retrieval Concepts , 2014, 2014 21st Asia-Pacific Software Engineering Conference.

[97]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[98]  Yu-Seung Ma Description of Method-level Mutation Operators for Java , 2011 .

[99]  Yves Le Traon,et al.  Sampling Program Inputs with Mutation Analysis: Going Beyond Combinatorial Interaction Testing , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[100]  Roseli A. Francelin Romero,et al.  Bayesian-Learning Based Guidelines to Determine Equivalent Mutants , 2001, Int. J. Softw. Eng. Knowl. Eng..

[101]  James M. Bieman,et al.  Mutation of Java objects , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[102]  Lionel C. Briand,et al.  Improving statechart testing criteria using data flow information , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[103]  Javier Tuya,et al.  Full predicate coverage for testing SQL database queries , 2010 .

[104]  René Just,et al.  MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[105]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[106]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[107]  James M. Bieman,et al.  A technique for mutation of Java objects , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[108]  Y. L. Traon,et al.  From genetic to bacteriological algorithms for mutation‐based testing , 2005, Softw. Test. Verification Reliab..

[109]  Roy P. Pargas,et al.  Mutation Testing of Software Using MIMD Computer , 1992, ICPP.

[110]  Sebastián Uchitel,et al.  Behaviour abstraction adequacy criteria for API call protocol testing , 2016, Softw. Test. Verification Reliab..

[111]  Kalpesh Kapoor,et al.  Formal Analysis of Coupling Hypothesis for Logical Faults , 2006, Innovations in Systems and Software Engineering.

[112]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[113]  Akbar Siami Namin,et al.  The influence of size and coverage on test suite effectiveness , 2009, ISSTA.

[114]  Ali Mesbah,et al.  Efficient JavaScript Mutation Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[115]  Darko Marinov,et al.  Balancing trade-offs in test-suite reduction , 2014, SIGSOFT FSE.

[116]  Bernhard K. Aichernig,et al.  MoMut::UML Model-Based Mutation Testing for UML , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[117]  Yves Le Traon,et al.  AjMutator: A Tool for the Mutation Analysis of AspectJ Pointcut Descriptors , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[118]  Dianxiang Xu,et al.  Prioritizing State-Based Aspect Tests , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[119]  Shing-Chi Cheung,et al.  Taming coincidental correctness: Coverage refinement with context patterns to improve fault localization , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[120]  Simeon C. Ntafos,et al.  ON TESTING WITH REQUIRED ELEMENTS. , 1981 .

[121]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[122]  Mark Hennessy,et al.  Analysing the effectiveness of rule-coverage as a reduction criterion for test suites of grammar-based software , 2008, Empirical Software Engineering.

[123]  Giovanni Denaro,et al.  Dynamic Data Flow Testing of Object Oriented Systems , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[124]  Fevzi Belli,et al.  Exploiting Model Morphology for Event-Based Testing , 2015, IEEE Transactions on Software Engineering.

[125]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[126]  A. Jefferson Offutt,et al.  Analyzing the validity of selective mutation with dominator mutants , 2016, SIGSOFT FSE.

[127]  Javier Tuya,et al.  Mutating database queries , 2007, Inf. Softw. Technol..

[128]  Nissim Francez,et al.  An Application of a Method for Analysis of Cyclic Prog rams , 1978, IEEE Transactions on Software Engineering.

[129]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[130]  K. S. How Tai Wah,et al.  An analysis of the coupling effect I: single test data , 2003, Sci. Comput. Program..

[131]  John A. Clark,et al.  Semantic Mutation Testing , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[132]  WahK. S. How Tai An analysis of the coupling effect I , 2003 .

[133]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[134]  Sebastián Uchitel,et al.  Behaviour Abstraction Coverage as Black-Box Adequacy Criteria , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[135]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[136]  Rui Abreu,et al.  On the empirical evaluation of similarity coefficients for spreadsheets fault localization , 2014, Automated Software Engineering.

[137]  Binh Nguyen Thanh,et al.  Survey on Mutation-based Test Data Generation , 2015 .

[138]  Christopher Wright,et al.  Mutation analysis of relational database schemas , 2015 .

[139]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[140]  Doo-Hwan Bae,et al.  Automated test case generation for FBD programs implementing reactor protection system software , 2014, Softw. Test. Verification Reliab..

[141]  Andres Flores,et al.  Testing‐based process for component substitutability , 2012, Softw. Test. Verification Reliab..

[142]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[143]  A. Jefferson Offutt The Coupling Effect: Fact or Fiction , 1989, Symposium on Testing, Analysis, and Verification.

[144]  Mark Harman,et al.  The relationship between program dependence and mutation analysis , 2001 .

[145]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[146]  Mats Per Erik Heimdahl,et al.  Better testing through oracle selection: (NIER track) , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[147]  Sarfraz Khurshid,et al.  An Empirical Study of JUnit Test-Suite Reduction , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[148]  Mike Papadakis,et al.  Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing , 2011, Software Quality Journal.

[149]  Jean-Marc Jézéquel,et al.  Genes and bacteria for automatic test cases optimization in the .NET environment , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[150]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[151]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[152]  Andres Flores,et al.  Testing-based assessment process for upgrading component systems , 2008, 2008 IEEE International Conference on Software Maintenance.

[153]  James Miller,et al.  A practical approach to testing GUI systems , 2006, Empirical Software Engineering.

[154]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[155]  Vahid Garousi,et al.  Automated Unit Testing of a SCADA Control Software: An Industrial Case Study Based on Action Research , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[156]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[157]  José Carlos Maldonado,et al.  Proteum/IM 2.0: An Integrated Mutation Testing Environment , 2001 .

[158]  Richard A. DeMillo Test Adequacy and Program Mutation , 1989, ICSE.

[159]  Gregg Rothermel,et al.  On-demand test suite reduction , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[160]  K. S. How Tai Wah,et al.  Fault coupling in finite bijective functions , 1995, Softw. Test. Verification Reliab..

[161]  Robert M. Hierons,et al.  SMT-C: A Semantic Mutation Testing Tools for C , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[162]  Alessandra Gorla,et al.  Search-based data-flow test generation , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[163]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[164]  Giuliano Antoniol,et al.  A case study using the round-trip strategy for state-based class testing , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[165]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[166]  Nikolai Tillmann,et al.  Test generation via Dynamic Symbolic Execution for mutation testing , 2010, 2010 IEEE International Conference on Software Maintenance.

[167]  Mike Papadakis,et al.  Automatic Mutation Test Case Generation via Dynamic Symbolic Execution , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[168]  Mohammad Zulkernine,et al.  Mutation-Based Testing of Format String Bugs , 2008, 2008 11th IEEE High Assurance Systems Engineering Symposium.

[169]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[170]  Adele E. Howe,et al.  Planner based error recovery testing , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[171]  Yves Le Traon,et al.  An approach for testing pointcut descriptors in AspectJ , 2011, Softw. Test. Verification Reliab..

[172]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999, Softw. Test. Verification Reliab..

[173]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[174]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[175]  Bernhard K. Aichernig,et al.  Killing strategies for model‐based mutation testing , 2015, Softw. Test. Verification Reliab..

[176]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[177]  Michel Delaunay,et al.  Towards Mutation Analysis for Lustre Programs , 2008, SLA++P@ETAPS.

[178]  Matthew Stephan Model Clone Detector Evaluation Using Mutation Analysis , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[179]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[180]  R. Lipton,et al.  Mutation analysis , 1998 .

[181]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[182]  Pearl Brereton,et al.  Lessons from applying the systematic literature review process within the software engineering domain , 2007, J. Syst. Softw..

[183]  K. S. How Tai Wah A Theoretical Study of Fault Coupling , 2000, Softw. Test. Verification Reliab..

[184]  Javier Tuya,et al.  Full predicate coverage for testing SQL database queries , 2010, Softw. Test. Verification Reliab..

[185]  Aditya P. Mathur,et al.  Interface Mutation: An Approach for Integration Testing , 2001, IEEE Trans. Software Eng..

[186]  Lionel C. Briand,et al.  Using simulation to empirically investigate test coverage criteria based on statechart , 2004, Proceedings. 26th International Conference on Software Engineering.

[187]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[188]  Laurie A. Williams,et al.  On guiding the augmentation of an automated test suite via mutation analysis , 2009, Empirical Software Engineering.

[189]  Gregg Rothermel,et al.  Oracle-Centric Test Case Prioritization , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[190]  Heung Seok Chae,et al.  An automated approach to reducing test suites for testing retargeted C compilers for embedded systems , 2011, J. Syst. Softw..

[191]  Ibrahim Habli,et al.  An Empirical Evaluation of Mutation Testing for Improving the Test Quality of Safety-Critical Software , 2013, IEEE Transactions on Software Engineering.

[192]  Gregg Rothermel,et al.  A Unified Test Case Prioritization Approach , 2014, ACM Trans. Softw. Eng. Methodol..

[193]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[194]  Yuhua Qi,et al.  Efficient Automated Program Repair through Fault-Recorded Testing Prioritization , 2013, 2013 IEEE International Conference on Software Maintenance.

[195]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[196]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.