An Analysis and Survey of the Development of Mutation Testing

Mutation Testing is a fault-based software testing technique that has been widely studied for over three decades. The literature on Mutation Testing has contributed a set of approaches, tools, developments, and empirical results. This paper provides a comprehensive analysis and survey of Mutation Testing. The paper also presents the results of several development trend analyses. These analyses provide evidence that Mutation Testing techniques and tools are reaching a state of maturity and applicability, while the topic of Mutation Testing itself is the subject of increasing interest.

[1]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[2]  Vi Andrew Jefferson Offutt,et al.  Automatic test data generation , 1988 .

[3]  J. Dingel,et al.  Comparative Assessment of Testing and Model Checking Using Program Mutation , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[4]  W. Eric Wong,et al.  Mutation Testing Applied to Validate Specifications Based on Petri Nets , 1995, FORTE.

[5]  Richard A. DeMillo,et al.  Compiler-integrated program mutation , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[6]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[7]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[8]  Byoungju Choi,et al.  Fault-Based Interface Testing Between Real-Time Operating System and Application , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[9]  M. R. Woodward.  Mutation testing-an evolving technique , 1990 .

[10]  Byoungju Choi,et al.  High-performance mutation testing , 1993, J. Syst. Softw..

[11]  Jr. Edward William Krauser.  Compiler-integrated software testing , 1992 .

[12]  M. Ellims,et al.  The Csaw C Mutation Tool: Initial Results , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[13]  J. A. Acree.  On mutation , 1980 .

[14]  William E. Riddle,et al.  Software technology maturation , 1985, ICSE '85.

[15]  Sudipto Ghosh,et al.  Interface Mutation to assess the adequacy of tests for components and systems , 2000, Proceedings. 34th International Conference on Technology of Object-Oriented Languages and Systems - TOOLS 34.

[16]  Roland H. Untch,et al.  Schema-based mutation analysis: a new test data adequacy assessment method , 1995 .

[17]  W. Eric Wong,et al.  Mutation Testing Applied to Validate SDL Specifications , 2004, TestCom.

[18]  Mark Harman,et al.  Multi Objective Higher Order Mutation Testing with Genetic Programming , 2009 .

[19]  Phyllis G. Frankl,et al.  Mutation Testing for Java Database Applications , 2009, 2009 International Conference on Software Testing Verification and Validation.

[20]  Y. Crouzet,et al.  The SESAME Experience: from Assembly Languages to Declarative Models , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[21]  Songtao Zhang,et al.  Lightweight State Based Mutation Testing for Security , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[22]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[23]  Sudipto Ghosh,et al.  Interface mutation , 2001, Softw. Test. Verification Reliab..

[24]  Auri Marcelo Rizzo Vincenzi,et al.  Unit and integration testing strategies for C programs using mutation , 2001, Softw. Test. Verification Reliab..

[25]  A. Jefferson Offutt,et al.  Generating test cases for web services using data perturbation , 2004, SOEN.

[26]  M. R. Woodward.  OBJTEST: an experimental testing tool for algebraic specifications , 1992 .

[27]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[28]  Mark Harman,et al.  Multi objective higher order mutation testing with GP , 2009, GECCO '09.

[29]  N. S. Barnett,et al.  Private communication , 1969 .

[30]  Pascale Thévenod-Fosse,et al.  A mutation analysis tool for Java programs , 2003, International Journal on Software Tools for Technology Transfer.

[31]  Giovanni Vigna,et al.  Testing network-based intrusion detection signatures using mutant exploits , 2004, CCS '04.

[32]  Roger T. Alexander,et al.  A Candidate Fault Model for AspectJ Pointcuts , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[33]  Richard H. Carver,et al.  Mutation-based testing of concurrent programs , 1993, Proceedings of IEEE International Test Conference - (ITC).

[34]  A. Jefferson Offutt.  The Coupling Effect: Fact or Fiction , 1989, Symposium on Testing, Analysis, and Verification.

[35]  Mohammad Zulkernine,et al.  MUSIC: Mutation-based SQL Injection Vulnerability Checking , 2008, 2008 The Eighth International Conference on Quality Software.

[36]  Akbar Siami Namin,et al.  Sufficient mutation operators for measuring test effectiveness , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[37]  Mark Harman,et al.  The relationship between program dependence and mutation analysis , 2001 .

[38]  Martin R. Woodward,et al.  An integrated system for program testing using weak mutation and data flow analysis , 1985, ICSE '85.

[39]  Jean-Marc Jézéquel,et al.  Trustable components: yet another mutation-based approach , 2000 .

[40]  K. S. How Tai Wah,et al.  Fault coupling in finite bijective functions , 1995, Softw. Test. Verification Reliab..

[41]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[42]  Gary McGraw,et al.  An automated approach for identifying potential vulnerabilities in software , 1998, Proceedings. 1998 IEEE Symposium on Security and Privacy (Cat. No.98CB36186).

[43]  R.A. DeMillo,et al.  Test Adequacy And Program Mutation , 1989, 11th International Conference on Software Engineering.

[44]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[45]  Y. Yesha,et al.  Specification mutation for test generation and analysis , 2004 .

[46]  Vernon Rego,et al.  High Performance Software Testing on SIMD Machines , 1991, IEEE Trans. Software Eng..

[47]  Akbar Siami Namin,et al.  On Sufficiency of Mutants , 2007, 29th International Conference on Software Engineering (ICSE'07 Companion).

[48]  J.H. Andrews,et al.  Finding Sufficient Mutation Operators via Variable Reduction , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[49]  M. R. Woodward,et al.  From weak to strong, dead or alive? an analysis of some mutation testing issues , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[50]  Yves Le Traon,et al.  Testing Security Policies: Going Beyond Functional Testing , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[51]  J. Dingel,et al.  ExMAn: A Generic and Customizable Framework for Experimental Mutation Analysis , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[52]  Tao Xie,et al.  A fault model and mutation testing of access control policies , 2007, WWW '07.

[53]  Simone do Rocio Senger de Souza,et al.  Mutation Testing Applied to Estelle Specifications , 2004, Software Quality Journal.

[54]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[55]  Mark Harman,et al.  Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet? , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[56]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[57]  Anna Derezinska.  Object-oriented mutation to assess the quality of tests , 2003, 2003 Proceedings 29th Euromicro Conference.

[58]  A. Jefferson Offutt,et al.  Generating test cases for XML-based Web component interactions using mutation analysis , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[59]  Jianping Wu,et al.  Mutation Testing of Protocol Messages Based on Extended TTCN-3 , 2008, 22nd International Conference on Advanced Information Networking and Applications (aina 2008).

[60]  Gregg Rothermel,et al.  A controlled experiment assessing test case prioritization techniques via mutation faults , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[61]  A. Jefferson Offutt,et al.  An experimental mutation system for Java , 2004, SOEN.

[62]  Paulo César Masiero,et al.  Mutation testing applied to validate specifications based on statecharts , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[63]  Jinhui Shan,et al.  An Approach to Test Data Generation for Killing Multiple Mutants , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[64]  Shing-Chi Cheung,et al.  Fault-based testing of database application programs with conceptual data model , 2005, Fifth International Conference on Quality Software (QSIC'05).

[65]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[66]  A. Jefferson Offutt,et al.  Mutation Operators for Ada , 1996 .

[67]  Mario Piattini,et al.  Integrating techniques and tools for testing automation , 2007, Softw. Test. Verification Reliab..

[68]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[69]  M. R. Woodward,et al.  Errors in algebraic specifications and an experimental mutation testing tool , 1993, Softw. Eng. J..

[70]  Susan Stepney,et al.  Challenging formal specifications by mutation: a CSP security example , 2003, Tenth Asia-Pacific Software Engineering Conference, 2003..

[71]  Benoit Baudry,et al.  A Generic Metamodel For Security Policies Mutation , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[72]  Vladimir N. Fleyshgakker,et al.  Efficient mutation analysis: a new approach , 1994, ISSTA '94.

[73]  Franco Fummi,et al.  A Mutation Model for the SystemC TLM 2.0 Communication Interfaces , 2008, 2008 Design, Automation and Test in Europe.

[74]  Anna Derezinska.  Quality Assessment of Mutation Operators Dedicated for C# Programs , 2006, 2006 Sixth International Conference on Quality Software (QSIC'06).

[75]  Brian Marick,et al.  The weak mutation hypothesis , 1991, TAV4.

[76]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[77]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[78]  L. du Bousquet,et al.  Mutation analysis for Lustre programs: Fault model description and validation , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[79]  A. Jefferson Offutt,et al.  An Experimental Evaluation of Data Flow and Mutation Testing , 1996 .

[80]  Christof J. Budnik,et al.  Basic Operations for Generating Behavioral Mutants , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[81]  Vincent Beroulle,et al.  Functional Verification of RTL Designs driven by Mutation Testing metrics , 2007, 10th Euromicro Conference on Digital System Design Architectures, Methods and Tools (DSD 2007).

[82]  Pascale Thévenod-Fosse,et al.  Software error analysis: a real case study involving real faults and mutations , 1996, ISSTA '96.

[83]  Mohammad Zulkernine,et al.  MUTEC: Mutation-based testing of Cross Site Scripting , 2009, 2009 ICSE Workshop on Software Engineering for Secure Systems.

[84]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[85]  Peter S. May,et al.  Test data generation : two evolutionary approaches to mutation testing , 2007 .

[86]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[87]  José Carlos Maldonado,et al.  Proteum/IM 2.0: An Integrated Mutation Testing Environment , 2001 .

[88]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[89]  Margo McCall,et al.  IEEE Computer Society , 2019, Encyclopedia of Software Engineering.

[90]  R. Lipton,et al.  Mutation analysis , 1998 .

[91]  Sudipto Ghosh,et al.  Testing component-based distributed applications , 2000 .

[92]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[93]  Rui Wang,et al.  Requirement Model-Based Mutation Testing for Web Service , 2008, 2008 4th International Conference on Next Generation Web Services Practices.

[94]  Eugene H. Spafford,et al.  Extending mutation testing to find environmental bugs , 1990, Softw. Pract. Exp..

[95]  Márcio Eduardo Delamaro,et al.  Proteum/FSM: a tool to support finite state machine validation based on mutation testing , 1999, Proceedings. SCCC'99 XIX International Conference of the Chilean Computer Science Society.

[96]  K. S. How Tai Wah.  A Theoretical Study of Fault Coupling , 2000, Softw. Test. Verification Reliab..

[97]  Roy P. Pargas,et al.  Mutation Testing of Software Using MIMD Computer , 1992, ICPP.

[98]  A. Hanks.  Canada , 2002 .

[99]  Kalpesh Kapoor,et al.  Formal Analysis of Coupling Hypothesis for Logical Faults , 2006, Innovations in Systems and Software Engineering.

[100]  Ivan Moore.  Jester - a JUnit test tester , 2001 .

[101]  Jean-Marc Jézéquel,et al.  Design by Contract to Improve Software Vigilance , 2006, IEEE Transactions on Software Engineering.

[102]  Richard J. Lipton,et al.  The design of a prototype mutation system for program testing , 1899, AFIPS National Computer Conference.

[103]  Aditya P. Mathur,et al.  Interface Mutation: An Approach for Integration Testing , 2001, IEEE Trans. Software Eng..

[104]  Xin Feng,et al.  ESTP: An Experimental Software Testing Platform , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[105]  A.P. Mathur.  Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[106]  Martin Erwig,et al.  Mutation Operators for Spreadsheets , 2009, IEEE Transactions on Software Engineering.

[107]  Ana R. Cavalli,et al.  Specification of Timed EFSM Fault Models in SDL , 2007, FORTE.

[108]  J F FULTON,et al.  Yale University , 1951, British medical journal.

[109]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[110]  M.J. Suarez-Cabal,et al.  SQLMutation: A tool to generate mutants of SQL database queries , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[111]  Aditya P. Mathur,et al.  Foundations of Software Testing , 2007 .

[112]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[113]  A. Jefferson Offutt,et al.  The Dynamic Domain Reduction Procedure for Test Data Generation: Design and Algorithms , 1994 .

[114]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[115]  A. Jefferson Offutt,et al.  The dynamic domain reduction procedure for test data generation , 1999, Softw. Pract. Exp..

[116]  Tao Xie,et al.  Efficient Mutant Generation for Mutation Testing of Pointcuts in Aspect-Oriented Programs , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[117]  Mohammad Zulkernine,et al.  Mutation-Based Testing of Buffer Overflow Vulnerabilities , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[118]  Gary McGraw,et al.  Software fault injection: inoculating programs against errors , 1997 .

[119]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[120]  Laurie A. Williams,et al.  On guiding the augmentation of an automated test suite via mutation analysis , 2009, Empirical Software Engineering.

[121]  Mario Piattini,et al.  Integrating techniques and tools for testing automation: Research Articles , 2007 .

[122]  Jeanne Marie Hanks.  Testing Cobol programs by mutation , 1980 .

[123]  Juan Luo,et al.  Testing Web services by XML perturbation , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[124]  Aditya P. Mathur,et al.  Integration testing using interface mutation , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[125]  John A. Clark,et al.  Class Mutation : Mutation Testing for Object-Oriented Programs , 2000 .

[126]  Bernhard K. Aichernig,et al.  Mutation Testing in the Refinement Calculus , 2003, Formal Aspects of Computing.

[127]  Philippe Chevalley,et al.  Applying mutation analysis for object-oriented programs using a reflective approach , 2001, Proceedings Eighth Asia-Pacific Software Engineering Conference.

[128]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[129]  A. Jefferson Offutt,et al.  How strong is weak mutation? , 1991, TAV4.

[130]  Giuliano Antoniol,et al.  Automatic mutation test input data generation via ant colony , 2007, GECCO '07.

[131]  Wenliang Du,et al.  Testing for software vulnerability using environment perturbation , 2000, Proceeding International Conference on Dependable Systems and Networks. DSN 2000.

[132]  Roland H. Untch,et al.  Mutation-based software testing using program schemata , 1992, ACM Southeast Regional Conference.

[133]  Paul A. Strooper,et al.  Mutation-based exploration of a method for verifying concurrent Java components , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[134]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[135]  Frederick C. Harris,et al.  Estimation and Enhancement of Real-Time Software Reliability Through Mutation Analysis , 1992, IEEE Trans. Computers.

[136]  W. Eric Wong,et al.  An Empirical Comparison of Mutation and Data Flow Based Test Adequacy Criteria , 1993 .

[137]  Byoungju Choi,et al.  Mutation-based inter-class testing , 1998, Proceedings 1998 Asia Pacific Software Engineering Conference (Cat. No.98EX240).

[138]  Tao Xie,et al.  Applying Interface-Contract Mutation in Regression Testing of Component-Based Software , 2007, 2007 IEEE International Conference on Software Maintenance.

[139]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[140]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[141]  Byoungju Choi.  Software testing using high-performance computers , 1991 .

[142]  A. Jefferson Offutt,et al.  Inter-class mutation operators for Java , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[143]  Anna Derezinska,et al.  Tool-Supported Advanced Mutation Approach for Verification of C# Programs , 2008, 2008 Third International Conference on Dependability of Computer Systems DepCoS-RELCOMEX.

[144]  M. Trakhtenbrot,et al.  New Mutations for Evaluation of Specification and Implementation Levels of Adequacy in Testing of Statecharts Models , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[145]  James Miller,et al.  Testing the semantics of W3C XML schema , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[146]  Tao Xie,et al.  Systematic Structural Testing of Firewall Policies , 2008, 2008 Symposium on Reliable Distributed Systems.

[147]  Yves Le Traon,et al.  A Test-Driven Approach to Developing Pointcut Descriptors in AspectJ , 2009, 2009 International Conference on Software Testing Verification and Validation.

[148]  Akihiko Tanaka.  Equivalence Testing for FORTRAN Mutation System Using Data Flow Analysis. , 1981 .

[149]  Yves Crouzet,et al.  An experimental study on software structural testing: deterministic versus random input generation , 1991, [1991] Digest of Papers. Fault-Tolerant Computing: The Twenty-First International Symposium.

[150]  John A. Clark,et al.  Investigating the effectiveness of object‐oriented testing strategies using the mutation method , 2001 .

[151]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second-order mutants , 2009 .

[152]  W. Eric Wong,et al.  Fault detection effectiveness of mutation and data flow testing , 1995, Software Quality Journal.

[153]  Aditya P. Mathur,et al.  Interface Mutation Test Adequacy Criterion: An Empirical Evaluation , 2001, Empirical Software Engineering.

[154]  Wenliang Du,et al.  Vulnerability Testing of Software System Using Fault Injection , 1999 .

[155]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[156]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[157]  A. Jefferson Offutt,et al.  Mutation-based testing criteria for timeliness , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[158]  R.A. DeMillo,et al.  An extended overview of the Mothra software testing environment , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[159]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[160]  Vladimir N. Fleyshgakker,et al.  Improved serial algorithms for mutation analysis , 1993, ISSTA '93.

[161]  Chanchal Kumar Roy,et al.  A Mutation/Injection-Based Automatic Framework for Evaluating Code Clone Detection Tools , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[162]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[163]  Tao Xie,et al.  Automated Generation of Pointcut Mutants for Testing Pointcuts in AspectJ Programs , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[164]  Yves Le Traon,et al.  AjMutator: A Tool for the Mutation Analysis of AspectJ Pointcut Descriptors , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[165]  Sudipto Ghosh,et al.  TDS: a tool for testing distributed component-based applications , 2001 .

[166]  M. Delaunay,et al.  Mutation Analysis for Reactive System Environment Properties , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[167]  Fabiano Cutigi Ferrari,et al.  Mutation Testing for Aspect-Oriented Programs , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[168]  Yong Rae Kwon,et al.  Evaluation of mutation testing for object-oriented programs , 2006, ICSE.

[169]  Mark Harman,et al.  Multi Objective Higher Order Mutation Testing with Genetic Programming , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[170]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: a case study in a space application , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[171]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[172]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[173]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[174]  B. Baudry,et al.  Mutation Analysis for Security Tests Qualification , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[175]  James M. Bieman,et al.  Mutation of Java objects , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[176]  Nikolai Tillmann,et al.  Mutation Analysis of Parameterized Unit Tests , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[177]  Yves Le Traon,et al.  Testing Security Policies: Going Beyond Functional Testing , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[178]  Yves Le Traon,et al.  Model-Based Tests for Access Control Policies , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[179]  Anna Derezinska.  Advanced mutation operators applicable in C# programs , 2006, SET.

[180]  Deepinder Sidhu,et al.  Fault coverage of protocol test methods , 1988, IEEE INFOCOM '88,Seventh Annual Joint Conference of the IEEE Computer and Communcations Societies. Networks: Evolution or Revolution?.

[181]  Jean-Marc Jézéquel,et al.  Genes and bacteria for automatic test cases optimization in the .NET environment , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[182]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[183]  Michel Delaunay,et al.  Using mutation analysis to evaluate test generation strategies in a synchronous context , 2007, International Conference on Software Engineering Advances (ICSEA 2007).

[184]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[185]  John A. Clark,et al.  Assessing Test Set Adequacy for Object-Oriented Programs Using Class Mutation , 2000 .

[186]  K. S. How Tai Wah,et al.  An analysis of the coupling effect I: single test data , 2003, Sci. Comput. Program..

[187]  Eric Wong,et al.  Testing for security vulnerabilities in software , 2012, ICSE 2012.

[188]  Vincent Beroulle,et al.  Impact of hardware emulation on the verification quality improvement , 2007, 2007 IFIP International Conference on Very Large Scale Integration.

[189]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[190]  John A. Clark,et al.  Search-based mutation testing for Simulink models , 2005, GECCO '05.

[191]  Richard J. Lipton,et al.  Theoretical and empirical studies on using program mutation to test the functional correctness of programs , 1980, POPL '80.

[192]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[193]  Bartosz Bogacki,et al.  Evaluation of Test Code Quality with Aspect-Oriented Mutations , 2006, XP.

[194]  Aditya P. Mathur,et al.  High performance testing on SIMD machines , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[195]  Bernhard K. Aichernig,et al.  From Faults Via Test Purposes to Test Cases: On the Fault-Based Testing of Concurrent Systems , 2006, FASE.

[196]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[197]  Mohammad Zulkernine,et al.  Mutation-Based Testing of Format String Bugs , 2008, 2008 11th IEEE High Assurance Systems Engineering Symposium.

[198]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[199]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[200]  Inmaculada Medina-Bulo,et al.  A Framework for Mutant Genetic Generation for WS-BPEL , 2009, SOFSEM.

[201]  Sudipto Ghosh,et al.  Towards measurement of testability of concurrent object-oriented programs using fault insertion: a preliminary investigation , 2002, Proceedings. Second IEEE International Workshop on Source Code Analysis and Manipulation.

[202]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999, Softw. Test. Verification Reliab..

[203]  Robert M. Hierons,et al.  Mutation testing from probabilistic and stochastic finite state machines , 2009, J. Syst. Softw..

[204]  Christian Norbert Zapf,et al.  MedusaMothra { A Distributed Interpreter for the Mothra Mutation Testing System , 1993 .

[205]  B.H. Smith,et al.  An Empirical Evaluation of the MuJava Mutation Operators , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[206]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[207]  Per Runeson,et al.  System level mutation analysis applied to a state-based language , 2001, Proceedings. Eighth Annual IEEE International Conference and Workshop On the Engineering of Computer-Based Systems-ECBS 2001.

[208]  Javier Tuya,et al.  Mutating database queries , 2007, Inf. Softw. Technol..

[209]  F. Wotawa,et al.  Mutant Minimization for Model-Checker Based Test-Case Generation , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[210]  J. Dingel,et al.  Mutation Operators for Concurrent Java (J2SE 5.0) , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[211]  Yinong Chen,et al.  Automatic Mutation Testing and Simulation on OWL-S Specified Web Services , 2008, 41st Annual Simulation Symposium (anss-41 2008).

[212]  J. Offutt,et al.  Mutation testing implements grammar-based testing , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[213]  Sang-Woon Kim,et al.  MUGAMMA: Mutation Analysis of Deployed Software to Increase Confidence and Assist Evolution , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[214]  Márcio Eduardo Delamaro,et al.  Interface mutation: assessing testing quality at interprocedural level , 1999, Proceedings. SCCC'99 XIX International Conference of the Chilean Computer Science Society.

[215]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[216]  Weichen Eric Wong.  On mutation and data flow , 1993 .

[217]  Baowen Xu,et al.  A Novel Method of Mutation Clustering Based on Domain Analysis , 2009, SEKE.

[218]  Morgan B Kaufmann,et al.  Mutation Testing for the New Century , 2002, Computer.

[219]  John A. Clark,et al.  The Rigorous Generation of Java Mutation Operators Using HAZOP , 1999 .

[220]  W. Eric Wong,et al.  An empirical comparison of data flow and mutation‐based test adequacy criteria , 1994, Softw. Test. Verification Reliab..

[221]  Jie Pan,et al.  Using Constraints to Detect Equivalent Mutants , 1994 .

[222]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[223]  K. N. King,et al.  A Fortran 77 interpreter for mutation analysis , 1987, PLDI 1987.

[224]  Márcio Eduardo Delamaro,et al.  Mutation analysis testing for finite state machines , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[225]  S. N. Weiss,et al.  All-Uses versus Mutation Testing : An ExperimentalComparison of E ectiveness , 1996 .

[226]  Inmaculada Medina-Bulo,et al.  GAmera: An Automatic Mutant Generation System for WS-BPEL Compositions , 2009, 2009 Seventh IEEE European Conference on Web Services.

[227]  Larry Joe Morell.  A theory of error-based testing , 1983 .

[228]  James M. Bieman,et al.  A technique for mutation of Java objects , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[229]  Márcio Eduardo Delamaro,et al.  Proteum - A Tool for the Assessment of Test Adequacy for C Programs User's guide , 1996 .

[230]  Timothy A. Budd,et al.  Program Testing by Specification Mutation , 1985, Comput. Lang..

[231]  Chanchal Kumar Roy,et al.  Towards a mutation-based automatic framework for evaluating code clone detection tools , 2008, C3S2E '08.

[232]  Jonas Mellin,et al.  Test Case Generation for Mutation-based Testing of Timeliness , 2006, Electron. Notes Theor. Comput. Sci..

[233]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[234]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[235]  Roberto da Silva Bigonha,et al.  A transformational language for mutant description , 2009, Comput. Lang. Syst. Struct..

[236]  David Jackson,et al.  Parallel firm mutation of Java programs , 2001 .

[237]  P. Black,et al.  Mutation of model checker specifications for test generation and evaluation , 2001 .

[238]  Bartosz Walter,et al.  Aspect-oriented Response Injection: an Alternative to Classical Mutation Testing , 2006, SET.

[239]  M. R. Woodward,et al.  Mutation testing - its origin and evolution , 1993, Inf. Softw. Technol..

[240]  R. Hierons,et al.  Mutation Testing from Probabilistic Finite State Machines , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[241]  Anna Derezińska,et al.  Cream - a system for object-oriented mutation of C# programs , 2007 .