Effective methods to tackle the equivalent mutant problem when testing software with mutation

Mutation Testing is undoubtedly one of the most effective software testing techniques that has been applied to different software artefacts at different testing levels. Apart from mutation’s versatility, its most important characteristic is its ability to detect real faults. Unfortunately, mutation’s adoption in practice is inhibited, primarily due to the manual effort involved in its application. This effort is attributed to the Equivalent Mutant Problem. The Equivalent Mutant Problem is a well-known impediment to mutation’s practical adoption that affects all phases of its application. To exacerbate the situation, the Equivalent Mutant Problem has been shown to be undecidable in its general form. Thus, no complete, automated solution exists. Although previous research has attempted to address this problem, its circumvention remains largely an open issue. This thesis argues that effective techniques that considerably ameliorate the problem’s adverse effects can be devised. To this end, the thesis introduces and empirically evaluates several such approaches that are based on Mutant Classification, Static Analysis and Code Similarity. First, the thesis proposes a novel mutant classification technique, named Isolating Equivalent Mutants (I-EQM) classifier, whose salient feature is the utilisation of second order mutants to automatically isolate first order equivalent ones. The empirical evaluation of the approach, based on real-world test subjects, suggests that I-EQM outperforms the existing techniques and results in a more effective testing process. Second, the thesis formally defines nine data flow patterns that can automatically detect equivalent and partially equivalent mutants. Their empirical evaluation

[1]  Alex Groce,et al.  MuCheck: an extensible tool for mutation testing of haskell programs , 2014, ISSTA 2014.

[2]  Frederick C. Harris,et al.  Estimation and Enhancement of Real-Time Software Reliability Through Mutation Analysis , 1992, IEEE Trans. Computers.

[3]  Nikolai Tillmann,et al.  Test generation via Dynamic Symbolic Execution for mutation testing , 2010, 2010 IEEE International Conference on Software Maintenance.

[4]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[5]  M.J. Suarez-Cabal,et al.  SQLMutation: A tool to generate mutants of SQL database queries , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[6]  Aditya P. Mathur,et al.  Foundations of Software Testing , 2007 .

[7]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[8]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[9]  Sarfraz Khurshid,et al.  Regression mutation testing , 2012, ISSTA 2012.

[10]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[11]  Mike Papadakis,et al.  Automatic Mutation Test Case Generation via Dynamic Symbolic Execution , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[12]  A.P. Mathur Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[13]  R. Lipton,et al.  Mutation analysis , 1998 .

[14]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[15]  Chanchal Kumar Roy,et al.  Comparison and evaluation of code clone detection techniques and tools: A qualitative approach , 2009, Sci. Comput. Program..

[16]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[17]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[18]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[19]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[20]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[21]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[22]  Daniel Jackson,et al.  A new model of program dependences for reverse engineering , 1994, SIGSOFT '94.

[23]  Maninder Singh,et al.  Software clone detection: A systematic review , 2013, Inf. Softw. Technol..

[24]  Yves Le Traon,et al.  Mutation-Based Generation of Software Product Line Test Configurations , 2014, SSBSE.

[25]  Sebastian Buchwald,et al.  Simple and Efficient Construction of Static Single Assignment Form , 2013, CC.

[26]  Mike Papadakis,et al.  Isolating First Order Equivalent Mutants via Second Order Mutation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[27]  Márcio Eduardo Delamaro,et al.  Mutation analysis testing for finite state machines , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[28]  Yves Le Traon,et al.  Mitigating the effects of equivalent mutants with mutant classification strategies , 2014, Sci. Comput. Program..

[29]  A. Jefferson Offutt,et al.  How strong is weak mutation? , 1991, TAV4.

[30]  Hiralal Agrawal,et al.  Dominators, super blocks, and program coverage , 1994, POPL '94.

[31]  Timothy J. Harvey,et al.  Practical improvements to the construction and destruction of static single assignment form , 1998 .

[32]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[33]  Timothy A. Budd,et al.  Program Testing by Specification Mutation , 1985, Comput. Lang..

[34]  K. S. How Tai Wah A Theoretical Study of Fault Coupling , 2000, Softw. Test. Verification Reliab..

[35]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[36]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[37]  Larry Joe Morell A theory of error-based testing , 1983 .

[38]  M. R. Woodward Mutation testing-an evolving technique , 1990 .

[39]  Roland H. Untch On reduced neighborhood mutation analysis using a single mutagenic operator , 2009, ACM-SE 47.

[40]  M. Ellims,et al.  The Csaw C Mutation Tool: Initial Results , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[41]  Yves Le Traon,et al.  Sound and Quasi-Complete Detection of Infeasible Test Requirements , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[42]  Ian Sommerville,et al.  Software engineering (6th ed.) , 2001 .

[43]  M. R. Woodward,et al.  Mutation testing - its origin and evolution , 1993, Inf. Softw. Technol..

[44]  R. Hierons,et al.  Mutation Testing from Probabilistic Finite State Machines , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[45]  Mark Harman,et al.  How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution , 2004, GECCO.

[46]  Milos Gligoric,et al.  SMutant: a tool for type-sensitive mutation testing in a dynamic language , 2011, ESEC/FSE '11.

[47]  J. A. Acree On mutation , 1980 .

[48]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[49]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[50]  Vi Andrew Jefferson Offutt,et al.  Automatic test data generation , 1988 .

[51]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[52]  Brian Marick,et al.  The weak mutation hypothesis , 1991, TAV4.

[53]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[54]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[55]  Weichen Eric Wong On mutation and data flow , 1993 .

[56]  Akbar Siami Namin,et al.  Sufficient mutation operators for measuring test effectiveness , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[57]  Yves Le Traon,et al.  Metallaxis‐FL: mutation‐based fault localization , 2015, Softw. Test. Verification Reliab..

[58]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[59]  Phyllis G. Frankl,et al.  Mutation Testing for Java Database Applications , 2009, 2009 International Conference on Software Testing Verification and Validation.

[60]  K. S. How Tai Wah,et al.  Fault coupling in finite bijective functions , 1995, Softw. Test. Verification Reliab..

[61]  R.A. DeMillo,et al.  Test Adequacy And Program Mutation , 1989, 11th International Conference on Software Engineering.

[62]  M. Wegman,et al.  Global value numbers and redundant computations , 1988, POPL '88.

[63]  Jean-Marc Jézéquel,et al.  Design by Contract to Improve Software Vigilance , 2006, IEEE Transactions on Software Engineering.

[64]  A. Jefferson Offutt,et al.  Detecting equivalent mutants and the feasible path problem , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[65]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[66]  Auri Marcelo Rizzo Vincenzi,et al.  Proteum: a family of tools to support specification and program testing based on mutation , 2001 .

[67]  William E. Howden Completeness criteria for testing elementary program functions , 1981, ICSE '81.

[68]  Mark N. Wegman,et al.  Efficiently computing static single assignment form and the control dependence graph , 1991, TOPL.

[69]  Macario Polo,et al.  Mutation at System and Functional Levels , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[70]  Jianping Wu,et al.  Mutation Testing of Protocol Messages Based on Extended TTCN-3 , 2008, 22nd International Conference on Advanced Information Networking and Applications (aina 2008).

[71]  A. Jefferson Offutt,et al.  Experimental Evaluation of SDL and One-Op Mutation for C , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[72]  Shuang Wang,et al.  Comparison of Unit-Level Automated Test Generation Tools , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[73]  John A. Clark,et al.  Investigating the effectiveness of object-oriented strategies with the mutation method (abstract only) , 2001 .

[74]  Larry J Morell,et al.  A Theory of Fault-Based Testing , 1990, IEEE Trans. Software Eng..

[75]  Mike Papadakis,et al.  Mutation based test case generation via a path selection strategy , 2012, Inf. Softw. Technol..

[76]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[77]  Hee Beng Kuan Tan,et al.  Heuristics-based infeasible path detection for dynamic test data generation , 2008, Inf. Softw. Technol..

[78]  Franz Wotawa,et al.  Using Constraints for Equivalent Mutant Detection , 2012, WS-FMDS.

[79]  Richard Torkar,et al.  Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation , 2014, IEEE Transactions on Software Engineering.

[80]  Bowen Alpern,et al.  Detecting equality of variables in programs , 1988, POPL '88.

[81]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[82]  Macario Polo,et al.  Mutation at the multi-class and system levels , 2013, Sci. Comput. Program..

[83]  Phil McMinn,et al.  The Impact of Equivalent, Redundant and Quasi Mutants on Database Schema Mutation Analysis , 2014, 2014 14th International Conference on Quality Software.

[84]  Andreas Zeller,et al.  Covering and Uncovering Equivalent Mutants , 2013, Softw. Test. Verification Reliab..

[85]  A. Jefferson Offutt,et al.  An Experimental Evaluation of Data Flow and Mutation Testing , 1996 .

[86]  Pascale Thévenod-Fosse,et al.  Software error analysis: a real case study involving real faults and mutations , 1996, ISSTA '96.

[87]  Auri Marcelo Rizzo Vincenzi,et al.  Toward the determination of sufficient mutant operators for C † , 2001, Softw. Test. Verification Reliab..

[88]  Phil McMinn,et al.  Efficient Mutation Analysis of Relational Database Structure Using Mutant Schemata and Parallelisation , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[89]  Atif M. Memon,et al.  Definition and evaluation of mutation operators for GUI-level mutation analysis , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[90]  David W. Binkley,et al.  Computing amorphous program slices using dependence graphs , 1999, SAC '99.

[91]  Nicos Malevris,et al.  The collateral coverage of data flow criteria when branch testing , 2006, Inf. Softw. Technol..

[92]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[93]  W. Eric Wong,et al.  An empirical comparison of data flow and mutation‐based test adequacy criteria , 1994, Softw. Test. Verification Reliab..

[94]  Kalpesh Kapoor,et al.  Formal Analysis of Coupling Hypothesis for Logical Faults , 2006, Innovations in Systems and Software Engineering.

[95]  Shing-Chi Cheung,et al.  Fault-based testing of database application programs with conceptual data model , 2005, Fifth International Conference on Quality Software (QSIC'05).

[96]  A. Jefferson Offutt,et al.  Empirical Evaluation of the Statement Deletion Mutation Operator , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[97]  Yves Le Traon,et al.  Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[98]  Phyllis G. Frankl,et al.  JDAMA: Java database application mutation analyser , 2011, Softw. Test. Verification Reliab..

[99]  Gary McGraw,et al.  Software fault injection: inoculating programs against errors , 1997 .

[100]  Nicos Malevris,et al.  MEDIC: A static analysis framework for equivalent mutant identification , 2015, Inf. Softw. Technol..

[101]  Frank Tip,et al.  Finding bugs efficiently with a SAT solver , 2007, ESEC-FSE '07.

[102]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[103]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[104]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[105]  Jacques Klein,et al.  Assessing Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[106]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[107]  Franz Wotawa,et al.  EqMutDetect — A tool for equivalent mutant detection in embedded systems , 2012, Proceedings of the 10th International Workshop on Intelligent Solutions in Embedded Systems.

[108]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[109]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[110]  B. Baudry,et al.  Mutation Analysis for Security Tests Qualification , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[111]  Nikolai Tillmann,et al.  Mutation Analysis of Parameterized Unit Tests , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[112]  Yves Le Traon,et al.  Testing Security Policies: Going Beyond Functional Testing , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[113]  René Just,et al.  The major mutation framework: efficient and scalable mutation analysis for Java , 2014, ISSTA 2014.

[114]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[115]  Yves Le Traon,et al.  Using Mutants to Locate "Unknown" Faults , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[116]  Douglas Baldwin,et al.  Heuristics for Determining Equivalence of Program Mutations. , 1979 .

[117]  Ali Mesbah,et al.  Efficient JavaScript Mutation Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[118]  Nicos Malevris,et al.  Identifying More Equivalent Mutants via Code Similarity , 2013, 2013 20th Asia-Pacific Software Engineering Conference (APSEC).

[119]  Richard J. Lipton,et al.  Theoretical and empirical studies on using program mutation to test the functional correctness of programs , 1980, POPL '80.

[120]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[121]  Martin P. Robillard,et al.  Tracking Code Clones in Evolving Software , 2007, 29th International Conference on Software Engineering (ICSE'07).

[122]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[123]  Mark Harman,et al.  A study of equivalent and stubborn mutation operators using human analysis of equivalence , 2014, ICSE.

[124]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[125]  R.A. DeMillo,et al.  An extended overview of the Mothra software testing environment , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[126]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[127]  Ali Mesbah,et al.  Guided Mutation Testing for JavaScript Web Applications , 2015, IEEE Transactions on Software Engineering.

[128]  A. Jefferson Offutt,et al.  Experiments with Data Flow and Mutation Testing , 1994 .

[129]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[130]  Shin Yoo,et al.  Ask the Mutants: Mutating Faulty Programs for Fault Localization , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[131]  Atif M. Memon,et al.  Conceptualization and Evaluation of Component-Based Testing Unified with Visual GUI Testing: An Empirical Study , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[132]  Elaine J. Weyuker,et al.  An Applicable Family of Data Flow Testing Criteria , 1988, IEEE Trans. Software Eng..

[133]  Mike Papadakis,et al.  Evaluating Mutation Testing Alternatives: A Collateral Experiment , 2010, 2010 Asia Pacific Software Engineering Conference.

[134]  Philippe Chevalley,et al.  Applying mutation analysis for object-oriented programs using a reflective approach , 2001, Proceedings Eighth Asia-Pacific Software Engineering Conference.

[135]  Sarfraz Khurshid,et al.  Operator-based and random mutant selection: Better together , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[136]  Yves Le Traon,et al.  Mutation testing strategies using mutant classification , 2013, SAC '13.

[137]  Lech Madeyski,et al.  Judy - a mutation testing tool for Java , 2010, IET Softw..

[138]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second-order mutants , 2009 .

[139]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[140]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[141]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[142]  Chanchal Kumar Roy,et al.  An Empirical Study of Function Clones in Open Source Software , 2008, 2008 15th Working Conference on Reverse Engineering.

[143]  Mike Papadakis,et al.  A Symbolic Execution Tool Based on the Elimination of Infeasible Paths , 2010, 2010 Fifth International Conference on Software Engineering Advances.

[144]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[145]  Brenda S. Baker,et al.  On finding duplication and near-duplication in large software systems , 1995, Proceedings of 2nd Working Conference on Reverse Engineering.

[146]  Elaine J. Weyuker,et al.  Data flow analysis techniques for test data selection , 2015, ICSE '82.

[147]  Nicos Malevris,et al.  Using Data Flow Patterns for Equivalent Mutant Detection , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[148]  A. Jefferson Offutt,et al.  Mutant Subsumption Graphs , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[149]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[150]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[151]  A Jeeerson Ooutt,et al.  Subsumption of Condition Coverage Techniques by Mutation Testing , 1996 .

[152]  Mike Papadakis,et al.  Mutation Testing Strategies - A Collateral Approach , 2010, ICSOFT.

[153]  Shinji Kusumoto,et al.  CCFinder: A Multilinguistic Token-Based Code Clone Detection System for Large Scale Source Code , 2002, IEEE Trans. Software Eng..

[154]  Akbar Siami Namin,et al.  On Sufficiency of Mutants , 2007, 29th International Conference on Software Engineering (ICSE'07 Companion).

[155]  M. R. Woodward,et al.  From weak to strong, dead or alive? an analysis of some mutation testing issues , 1988, [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis.

[156]  Yves Le Traon,et al.  Testing Security Policies: Going Beyond Functional Testing , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[157]  Tao Xie,et al.  A fault model and mutation testing of access control policies , 2007, WWW '07.

[158]  Jong-Deok Choi,et al.  Automatic construction of sparse data flow evaluation graphs , 1991, POPL '91.

[159]  A. Jefferson Offutt The Coupling Effect: Fact or Fiction , 1989, Symposium on Testing, Analysis, and Verification.

[160]  Mark Harman,et al.  The relationship between program dependence and mutation analysis , 2001 .

[161]  Martin R. Woodward,et al.  An integrated system for program testing using weak mutation and data flow analysis , 1985, ICSE '85.

[162]  Mark Harman,et al.  Amorphous program slicing , 2003, J. Syst. Softw..

[163]  J.H. Andrews,et al.  Finding Sufficient Mutation Operators via Variable Reduction , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[164]  Aditya P. Mathur,et al.  Integration testing using interface mutation , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[165]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[166]  Mike Papadakis,et al.  Employing second‐order mutation for isolating first‐order equivalent mutants , 2015, Softw. Test. Verification Reliab..

[167]  Anna Derezinska,et al.  Experimental Evaluation of Mutation Testing Approaches to Python Programs , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[168]  Javier Tuya,et al.  Mutating database queries , 2007, Inf. Softw. Technol..

[169]  Tao Xie,et al.  Is operator-based mutant selection superior to random mutant selection? , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[170]  Nicos Malevris A path generation method for testing LCSAJs that restrains infeasible paths , 1995, Inf. Softw. Technol..