HMER: A Hybrid Mutation Execution Reduction approach for Mutation-based Fault Localization

Abstract Identifying the location of faults in programs has been recognized as one of the most manually and time cost activities during software debugging process. Fault localization techniques, which seek to identify faulty program statements as quickly as possible, can assist developers in alleviating the time and manual cost of software debugging. Mutation-based fault localization(MBFL) has a promising fault localization accuracy, but suffered from huge mutation execution cost. To reduce the cost of MBFL, we propose a Hybrid Mutation Execution Reduction(HMER) approach in this paper. HMER consists of two steps: Weighted Statement-Oriented Mutant Sampling(WSOME) and Dynamic Mutation Execution Strategy(DMES). In the first step, we employ Spectrum-Based Fault Localization(SBFL) techniques to calculate the suspiciousness value of statements, and guarantee that the mutants generated from statements with higher suspiciousness value will have more chance to be remained in the sampling process. Next, a dynamic mutation execution strategy is used to execute the reduced mutant set on test suite to avoid worthless execution. Empirical results on 130 versions from 9 subject programs show that HMER can reduce 74.5%-93.4% mutation execution cost while keeping almost the same fault localization accuracy with the original MBFL. A further W i l c o x o n s i g n e d − r a n k t e s t indicates that when employing HMER strategy in MBFL, the fault localization accuracy has no statistically significant difference in most cases compared with the original MBFL without any reduction techniques.

[1]  Joseph Robert Horgan,et al.  Fault localization using execution slices and dataflow tests , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[2]  Yong Liu,et al.  An optimal mutation execution strategy for cost reduction of mutation-based fault localization , 2018, Inf. Sci..

[3]  Stephen M. Thebaut,et al.  An approach to software fault localization and revalidation based on incremental data flow analysis , 1989, [1989] Proceedings of the Thirteenth Annual International Computer Software & Applications Conference.

[4]  Giorgio Di Natale,et al.  Computing reliability: On the differences between software testing and software fault injection techniques , 2017, Microprocess. Microsystems.

[5]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[6]  Michael D. Ernst,et al.  Evaluating and Improving Fault Localization , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[7]  Alessandro Oliveira Arantes,et al.  On Proposing a Test Oracle Generator Based on Static and Dynamic Source Code Analysis , 2015, QRS Companion.

[8]  William E. Howden,et al.  Theoretical and Empirical Studies of Program Testing , 1978, IEEE Transactions on Software Engineering.

[9]  W. Eric Wong,et al.  The DStar Method for Effective Software Fault Localization , 2014, IEEE Transactions on Reliability.

[10]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[11]  Kai-Yuan Cai,et al.  Exploring the usefulness of unlabelled test cases in software fault localization , 2018, J. Syst. Softw..

[12]  James R. Larus,et al.  The use of program profiling for software maintenance with applications to the year 2000 problem , 1997, ESEC '97/FSE-5.

[13]  Yves Le Traon,et al.  Effective fault localization via mutation analysis: a selective mutation approach , 2014, SAC.

[14]  A. Jefferson Offutt,et al.  Test Oracle Strategies for Model-Based Testing , 2017, IEEE Transactions on Software Engineering.

[15]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[16]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[17]  Lars Grunske,et al.  An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems , 2019, Softw. Pract. Exp..

[18]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[19]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[20]  Zheng Li,et al.  Identify Coincidental Correct Test Cases Based on Fuzzy Classification , 2016, 2016 International Conference on Software Analysis, Testing and Evolution (SATE).

[21]  James A. Jones,et al.  On the influence of multiple faults on coverage-based fault localization , 2011, ISSTA '11.

[22]  Tao Wang,et al.  Automated path generation for software fault localization , 2005, ASE '05.

[23]  Wes Masri,et al.  An empirical study of the factors that reduce the effectiveness of coverage-based fault localization , 2009, DEFECTS '09.

[24]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[25]  Qian Yang,et al.  A Survey of Coverage-Based Testing Tools , 2009, Comput. J..

[26]  Zuohua Ding,et al.  Fault localization based on statement frequency , 2016, Inf. Sci..

[27]  René Just,et al.  The major mutation framework: efficient and scalable mutation analysis for Java , 2014, ISSTA 2014.

[28]  Hadi Hemmati,et al.  Studying Test Case Failure Prediction for Test Case Prioritization , 2017, PROMISE.

[29]  Chao Liu,et al.  Statistical Debugging: A Hypothesis Testing-Based Approach , 2006, IEEE Transactions on Software Engineering.

[30]  Yves Le Traon,et al.  Metallaxis‐FL: mutation‐based fault localization , 2015, Softw. Test. Verification Reliab..

[31]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[32]  Peter Zoeteweij,et al.  An Evaluation of Similarity Coefficients for Software Fault Localization , 2006, 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06).

[33]  José Carlos Maldonado,et al.  Proteum/IM 2.0: An Integrated Mutation Testing Environment , 2001 .

[34]  Sarfraz Khurshid,et al.  Faster mutation testing inspired by test prioritization and reduction , 2013, ISSTA.

[35]  Wes Masri,et al.  Prevalence of coincidental correctness and mitigation of its impact on fault localization , 2014, TSEM.

[36]  Stephen H. Edwards,et al.  Using Spectrum-Based Fault Location and Heatmaps to Express Debugging Suggestions to Student Programmers , 2017, ACE '17.

[37]  Mark Harman,et al.  Predictive Mutation Testing , 2019, IEEE Transactions on Software Engineering.

[38]  W. Eric Wong,et al.  Combining mutation and fault localization for automated program debugging , 2014, J. Syst. Softw..

[39]  Lars Grunske,et al.  A Critical Evaluation of Spectrum-Based Fault Localization Techniques on a Large-Scale Software System , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[40]  Yuriy Brun,et al.  Finding latent code errors via machine learning over program executions , 2004, Proceedings. 26th International Conference on Software Engineering.

[41]  Wes Masri,et al.  Fault localization based on information flow coverage , 2010, Softw. Test. Verification Reliab..

[42]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[43]  Yu Qi,et al.  Effective program debugging based on execution slices and inter-block data dependency , 2006, J. Syst. Softw..

[44]  Doo-Hwan Bae,et al.  A Theoretical Framework for Understanding Mutation-Based Testing Methods , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[45]  Auri Marcelo Rizzo Vincenzi,et al.  FTMES: A Failed-Test-Oriented Mutant Execution Strategy for Mutation-Based Fault Localization , 2018, ISSRE.

[46]  Lee Naish,et al.  A model for spectra-based software diagnosis , 2011, TSEM.

[47]  Andreas Zeller,et al.  Lightweight bug localization with AMPLE , 2005, AADEBUG'05.

[48]  Gregg Rothermel,et al.  An empirical investigation of program spectra , 1998, PASTE '98.

[49]  Markus Stumptner,et al.  Locating Bugs in Java Programs - First Results of the Java Diagnosis Experiment Project , 2000, IEA/AIE.

[50]  Yves Le Traon,et al.  Assessing and Comparing Mutation-based Fault Localization Techniques , 2016, ArXiv.

[51]  Shin Hong,et al.  MUSEUM: Debugging real-world multilingual programs using mutation analysis , 2017, Inf. Softw. Technol..

[52]  Chao Liu,et al.  Using Cluster Analysis to Identify Coincidental Correctness in Fault Localization , 2012, 2012 Fourth International Conference on Computational and Information Sciences.

[53]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[54]  Yan Shi,et al.  Using an RBF Neural Network to Locate Program Bugs , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[55]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[56]  Yong Liu,et al.  Statement-Oriented Mutant Reduction Strategy for Mutation Based Fault Localization , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[57]  Olivier Ridoux,et al.  Formal Concept Analysis Enhances Fault Localization in Software , 2008, ICFCA.

[58]  Xiangyu Zhang,et al.  Experimental evaluation of using dynamic slices for fault location , 2005, AADEBUG'05.

[59]  Gregg Rothermel,et al.  An empirical investigation of the relationship between spectra differences and regression faults , 2000 .

[60]  Baowen Xu,et al.  Code Coverage-Based Failure Proximity without Test Oracles , 2016, 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC).

[61]  Yves Le Traon,et al.  Using Mutants to Locate "Unknown" Faults , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.