Mutant reduction based on dominance relation for weak mutation testing

Context: As a fault-based testing technique, mutation testing is effective at evaluating the quality of existing test suites. However, a large number of mutants result in the high computational cost in mutation testing. As a result, mutant reduction is of great importance to improve the efficiency of mutation testing.Objective: We aim to reduce mutants for weak mutation testing based on the dominance relation between mutant branches.Method: In our method, a new program is formed by inserting mutant branches into the original program. By analyzing the dominance relation between mutant branches in the new program, the non-dominated one is obtained, and the mutant corresponding to the non-dominated mutant branch is the mutant after reduction.Results: The proposed method is applied to test ten benchmark programs and six classes from open-source projects. The experimental results show that our method reduces over 80% mutants on average, which greatly improves the efficiency of mutation testing.Conclusion: We conclude that dominance relation between mutant branches is very important and useful in reducing mutants for mutation testing.

[1]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[2]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[3]  A. Jefferson Offutt,et al.  An Experimental Evaluation of Data Flow and Mutation Testing , 1996 .

[4]  Timothy Alan Budd,et al.  Mutation analysis of program test data , 1980 .

[5]  Antonia Bertolino,et al.  Automatic Generation of Path Covers Based on the Control Flow Analysis of Computer Programs , 1994, IEEE Trans. Software Eng..

[6]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[7]  Ahmed S. Ghiduk,et al.  Higher order mutation testing: A Systematic Literature Review , 2017, Comput. Sci. Rev..

[8]  Joachim Wegener,et al.  Evaluating Evolutionary Testability with Software-Measurements , 2004, GECCO.

[9]  Inmaculada Medina-Bulo,et al.  Evolutionary mutation testing , 2011, Inf. Softw. Technol..

[10]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[11]  Mark Harman,et al.  Constructing Subtle Faults Using Higher Order Mutation Testing , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[12]  A.P. Mathur.  Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[13]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[14]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[15]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2012, IEEE Trans. Software Eng..

[16]  A. Jefferson Offutt,et al.  How strong is weak mutation? , 1991, TAV4.

[17]  Nikolai Tillmann,et al.  Test generation via Dynamic Symbolic Execution for mutation testing , 2010, 2010 IEEE International Conference on Software Maintenance.

[18]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[19]  Mark Harman,et al.  Multi Objective Higher Order Mutation Testing with Genetic Programming , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[20]  René Just,et al.  Using Non-redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.

[21]  A. Jefferson Offutt,et al.  Better predicate testing , 2011, AST '11.

[22]  René Just,et al.  Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[23]  Sergio Segura,et al.  Automated metamorphic testing on the analyses of feature models , 2011, Inf. Softw. Technol..

[24]  Hu Ji.  Research on Software Fault Injection Testing , 2015 .

[25]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[26]  Jinfu Chen,et al.  Research on Software Fault Injection Testing: Research on Software Fault Injection Testing , 2009 .

[27]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[28]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999, Softw. Test. Verification Reliab..

[29]  B.H. Smith,et al.  An Empirical Evaluation of the MuJava Mutation Operators , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[30]  Leonardo Bottaci,et al.  Efficiency of mutation operators and selective mutation strategies: an empirical study , 1999 .

[31]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[32]  Michael D. Ernst,et al.  Using State Infection Conditions to Detect Equivalent Mutants and Speed up Mutation Analysis , 2013, ArXiv.

[33]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[34]  J. A. Bondy,et al.  Graph Theory with Applications , 1978 .

[35]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[36]  J. A. Bondy,et al.  Graph Theory with Applications , 1978 .

[37]  Mark Harman,et al.  Multi Objective Higher Order Mutation Testing with Genetic Programming , 2009 .

[38]  Mike Papadakis,et al.  Mutation based test case generation via a path selection strategy , 2012, Inf. Softw. Technol..

[39]  D. Gong,et al.  Automatic detection of infeasible paths in software testing , 2010, IET Softw..

[40]  Mario Piattini,et al.  Decreasing the cost of mutation testing with second-order mutants , 2009 .

[41]  A. Jefferson Offutt,et al.  An Experimental Evaluation of Data Flow and Mutation Testing , 1996, Softw. Pract. Exp..

[42]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[43]  Ming-Hao Liu,et al.  A New Approach to Automated Test Data Generation in Mutation Testing: A New Approach to Automated Test Data Generation in Mutation Testing , 2009 .

[44]  A. Jefferson Offutt,et al.  Static analysis of mutant subsumption , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[45]  Shan Jin.  A New Approach to Automated Test Data Generation in Mutation Testing , 2008 .

[46]  Ahmed S. Ghiduk,et al.  Using Genetic Algorithms and Dominance Concepts for Generating Reduced Test Data , 2010, Informatica.

[47]  Antonia Bertolino,et al.  Using Spanning Sets for Coverage Testing , 2003, IEEE Trans. Software Eng..

[48]  Lu Yansheng,et al.  Research on Software Fault Injection Testing , 2009 .

[49]  Mike Papadakis,et al.  Evaluating Mutation Testing Alternatives: A Collateral Experiment , 2010, 2010 Asia Pacific Software Engineering Conference.

[50]  Mark Harman,et al.  A study of equivalent and stubborn mutation operators using human analysis of equivalence , 2014, ICSE.

[51]  A. Jefferson Offutt,et al.  Establishing Theoretical Minimal Sets of Mutants , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[52]  Mike Papadakis,et al.  Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing , 2011, Software Quality Journal.

[53]  Nicos Malevris,et al.  Using Data Flow Patterns for Equivalent Mutant Detection , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[54]  A. Jefferson Offutt,et al.  Mutant Subsumption Graphs , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[55]  Man F. Lau,et al.  An extended fault class hierarchy for specification-based testing , 2005, TSEM.