Metallaxis‐FL: mutation‐based fault localization

Fault localization methods seek to identify faulty program statements based on the information provided by the failing and passing test executions. Spectrum‐based methods are among the most popular ones and assist programmers by assigning suspiciousness values on program statements according to their probability of being faulty. This paper proposes Metallaxis, a fault localization approach based on mutation analysis. The innovative part of Metallaxis is that it uses mutants and links them with the faulty program places. Thus, mutants that are killed mostly by failing tests provide a good indication about the location of a fault. Experimentation using Metallaxis suggests that it is significantly more effective than statement‐based approaches. This is true even in the case where mutation cost‐reduction techniques, such as mutant sampling, are facilitated. Additionally, results from a controlled experiment show that the use of mutation as a testing technique provides benefits to the fault localization process. Therefore, fault localization is significantly improved by using mutation‐based tests instead of block‐based or branch‐based test suites. Finally, evidence in support of the methods’ scalability is also given. Copyright © 2013 John Wiley & Sons, Ltd.

[1]  Xiangyu Zhang,et al.  Locating faults using multiple spectra-specific models , 2011, SAC.

[2]  W. Eric Wong,et al.  Using Mutation to Automatically Suggest Fixes for Faulty Programs , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[3]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[4]  Mike Papadakis,et al.  An Empirical Evaluation of the First and Second Order Mutation Testing Strategies , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[5]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[6]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[7]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[8]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[9]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[10]  Xiangyu Zhang,et al.  Pruning dynamic slices with confidence , 2006, PLDI '06.

[11]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[12]  Michael D. Ernst,et al.  Improving test suites via operational abstraction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[13]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[14]  Chao Liu,et al.  SOBER: statistical model-based bug localization , 2005, ESEC/FSE-13.

[15]  David Notkin,et al.  Editorial—looking back , 2013, TSEM.

[16]  James H. Andrews,et al.  Evaluating the Accuracy of Fault Localization Techniques , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[17]  Mike Papadakis,et al.  Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing , 2011, Software Quality Journal.

[18]  Alessandro Orso,et al.  Are automated debugging techniques actually helping programmers? , 2011, ISSTA '11.

[19]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[20]  Wes Masri,et al.  Fault localization based on information flow coverage , 2010, Softw. Test. Verification Reliab..

[21]  Yves Le Traon,et al.  Improving test suites for efficient fault localization , 2006, ICSE.

[22]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[23]  Peter Zoeteweij,et al.  Spectrum-Based Multiple Fault Localization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[24]  Rajiv Gupta,et al.  Fault localization using value replacement , 2008, ISSTA '08.

[25]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[26]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[27]  Auri Marcelo Rizzo Vincenzi,et al.  Proteum: a family of tools to support specification and program testing based on mutation , 2001 .

[28]  Mike Papadakis,et al.  Automatic Mutation Test Case Generation via Dynamic Symbolic Execution , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[29]  Byoungju Choi,et al.  A family of code coverage-based heuristics for effective fault localization , 2010, J. Syst. Softw..

[30]  H. Cleve,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[31]  Mike Papadakis,et al.  Mutation based test case generation via a path selection strategy , 2012, Inf. Softw. Technol..

[32]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[33]  Morgan B Kaufmann,et al.  Mutation Testing for the New Century , 2002, Computer.

[34]  Franz Wotawa,et al.  On the use of mutations and testing for debugging , 2013, Softw. Pract. Exp..

[35]  Andy Podgurski,et al.  The Probabilistic Program Dependence Graph and Its Application to Fault Diagnosis , 2008, IEEE Transactions on Software Engineering.

[36]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[37]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[38]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[39]  Mike Papadakis,et al.  Isolating First Order Equivalent Mutants via Second Order Mutation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[40]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[41]  Martin Burger,et al.  Minimizing reproduction of software failures , 2011, ISSTA '11.

[42]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[43]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[44]  Mark Harman,et al.  Fault localization prioritization: Comparing information-theoretic and coverage-based approaches , 2013, TSEM.

[45]  Yves Le Traon,et al.  Using Mutants to Locate "Unknown" Faults , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[46]  Xiangyu Zhang,et al.  Whole execution traces and their applications , 2005, TACO.

[47]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[48]  Michael R. Lyu,et al.  A coverage analysis tool for the effectiveness of software testing , 1993, Proceedings of 1993 IEEE International Symposium on Software Reliability Engineering.