Performance Analysis of Maximal Risk Evaluation Formulas for Spectrum-Based Fault Localization

The effectiveness analysis of risk evaluation formulas has become a significant research area in spectrum-based fault localization (SBFL). The risk evaluation formula is designed and widely used to evaluate the likelihood of a program spectrum to be faulty. There are numerous empirical and theoretical studies to investigate and compare the performance between sixty risk evaluation formulas. According to previous research, these sixty risk evaluation formulas together form a partially ordered set. Among them, nine formulas are maximal. These nine formulas can further be grouped into five maximal risk evaluation formula groups so that formulas in the same group have the same performance. Moreover, previous research showed that we cannot theoretically compare formulas across these five maximal formula groups. However, experimental data “suggests” that a maximal formula in one group could outperform another one (from a different group) more frequently, though not always. This inspired us to further investigate the performance between any two maximal formulas in different maximal formula groups. Our approach involves two major steps. First, we propose a new condition to compare between two different maximal formulas. Based on this new condition, we present five different scenarios under which a formula performs better than another. This is different from the condition suggested by the previous theoretical study. We performed an empirical study to compare different maximal formulas using our condition. Our results showed that among two maximal risk evaluation formulas, it is feasible to identify one that can outperform the others more frequently.

[1]  Byoungju Choi,et al.  A family of code coverage-based heuristics for effective fault localization , 2010, J. Syst. Softw..

[2]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[3]  Mark Harman,et al.  Human Competitiveness of Genetic Programming in Spectrum-Based Fault Localisation , 2017, ACM Trans. Softw. Eng. Methodol..

[4]  John Stasko,et al.  Visualization for Fault Localization , 2003 .

[5]  Joseph Robert Horgan,et al.  Fault localization using execution slices and dataflow tests , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[6]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[7]  Michael D. Ernst,et al.  Evaluating and Improving Fault Localization , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[8]  Bo Jiang,et al.  On the Integration of Test Adequacy, Test Case Prioritization, and Statistical Fault Localization , 2010, 2010 10th International Conference on Quality Software.

[9]  P. F. Russell,et al.  On Habitat and Association of Species of Anopheline Larvae in South-eastern Madras. , 1940 .

[10]  Peter Zoeteweij,et al.  An Evaluation of Similarity Coefficients for Software Fault Localization , 2006, 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06).

[11]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[12]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[13]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[14]  Lee Naish,et al.  A model for spectra-based software diagnosis , 2011, TSEM.

[15]  Baowen Xu,et al.  A Revisit of a Theoretical Analysis on Spectrum-Based Fault Localization , 2015, 2015 IEEE 39th Annual Computer Software and Applications Conference.

[16]  Baowen Xu,et al.  A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization , 2013, TSEM.

[17]  Tsong Yueh Chen,et al.  How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[18]  Gregg Rothermel,et al.  An empirical investigation of the relationship between spectra differences and regression faults , 2000 .

[19]  Baowen Xu,et al.  On the analysis of spectrum based fault localization using hitting sets , 2019, J. Syst. Softw..

[20]  Yu Qi,et al.  Effective program debugging based on execution slices and inter-block data dependency , 2006, J. Syst. Softw..

[21]  Mark Harman,et al.  No pot of gold at the end of program spectrum rainbow: Greatest risk evaluation formula does not exist , 2014 .

[22]  Yuhua Qi,et al.  Slice-based statistical fault localization , 2014, J. Syst. Softw..

[23]  Lee Naish,et al.  Study of the relationship of bug consistency with respect to performance of spectra metrics , 2009, 2009 2nd IEEE International Conference on Computer Science and Information Technology.

[24]  Mark Harman,et al.  Fault localization prioritization: Comparing information-theoretic and coverage-based approaches , 2013, TSEM.

[25]  Yu Qi,et al.  Smart debugging software architectural design in SDL , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[26]  Shin Yoo,et al.  Evolving Human Competitive Spectra-Based Fault Localisation Techniques , 2012, SSBSE.

[27]  Kai-Yuan Cai,et al.  Effective Fault Localization using Code Coverage , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[28]  Baowen Xu,et al.  Metamorphic slice: An application in spectrum-based fault localization , 2013, Inf. Softw. Technol..

[29]  T. H. Tse,et al.  On Practical Adequate Test Suites for Integrated Test Case Prioritization and Fault Localization , 2011, 2011 11th International Conference on Quality Software.

[30]  Lei Zhao,et al.  A Crosstab-based Statistical Method for Effective Fault Localization , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[31]  Kai-Yuan Cai,et al.  Exploring the usefulness of unlabelled test cases in software fault localization , 2018, J. Syst. Softw..

[32]  Mark Harman,et al.  Provably Optimal and Human-Competitive Results in SBSE for Spectrum Based Fault Localisation , 2013, SSBSE.

[33]  Yanping Chen,et al.  Specification-based regression test selection with risk analysis , 2002, CASCON.

[34]  Gregg Rothermel,et al.  An empirical investigation of program spectra , 1998, PASTE '98.

[35]  Zheng Zheng,et al.  Robustness of spectrum-based fault localisation in environments with labelling perturbations , 2019, J. Syst. Softw..

[36]  Baowen Xu,et al.  Spectrum-Based Fault Localization: Testing Oracles are No Longer Mandatory , 2011, 2011 11th International Conference on Quality Software.

[37]  Peter Zoeteweij,et al.  A practical evaluation of spectrum-based fault localization , 2009, J. Syst. Softw..

[38]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.