Should I follow this fault localization tool’s output?

Debugging is a crucial yet expensive activity to improve the reliability of software systems. To reduce debugging cost, various fault localization tools have been proposed. A spectrum-based fault localization tool often outputs an ordered list of program elements sorted based on their likelihood to be the root cause of a set of failures (i.e., their suspiciousness scores). Despite the many studies on fault localization, unfortunately, however, for many bugs, the root causes are often low in the ordered list. This potentially causes developers to distrust fault localization tools. Recently, Parnin and Orso highlight in their user study that many debuggers do not find fault localization useful if they do not find the root cause early in the list. To alleviate the above issue, we build an oracle that could predict whether the output of a fault localization tool can be trusted or not. If the output is not likely to be trusted, developers do not need to spend time going through the list of most suspicious program elements one by one. Rather, other conventional means of debugging could be performed. To construct the oracle, we extract the values of a number of features that are potentially related to the effectiveness of fault localization. Building upon advances in machine learning, we process these feature values to learn a discriminative model that is able to predict the effectiveness of a fault localization tool output. In this work, we consider an output of a fault localization tool to be effective if the root cause appears in the top 10 most suspicious program elements. We have evaluated our proposed oracle on 200 faulty versions of Space, NanoXML, XML-Security, and the 7 programs in Siemens test suite. Our experiments demonstrate that we could predict the effectiveness of 9 fault localization tools with a precision, recall, and F-measure (harmonic mean of precision and recall) of up to 74.38 %, 90.00 % and 81.45 %, respectively. The numbers indicate that many ineffective fault localization instances are identified correctly, while only few effective ones are identified wrongly.

[1]  Frank Tip,et al.  Directed test generation for effective fault localization , 2010, ISSTA '10.

[2]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[3]  Ken-ichi Matsumoto,et al.  Predicting Re-opened Bugs: A Case Study on the Eclipse Project , 2010, 2010 17th Working Conference on Reverse Engineering.

[4]  Dianxiang Xu,et al.  Towards Better Fault Localization: A Crosstab-Based Statistical Approach , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[5]  Hong Cheng,et al.  Bug Signature Minimization and Fusion , 2011, 2011 IEEE 13th International Symposium on High-Assurance Systems Engineering.

[6]  Jin Liu,et al.  Dictionary learning based software defect prediction , 2014, ICSE.

[7]  David Lo,et al.  Search-based fault localization , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[8]  Nicholas Jalbert,et al.  Automated duplicate detection for bug tracking systems , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[9]  David Lo,et al.  Comprehensive evaluation of association measures for fault localization , 2010, 2010 IEEE International Conference on Software Maintenance.

[10]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[11]  Xiangyu Zhang,et al.  Locating faulty code using failure-inducing chops , 2005, ASE.

[12]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[13]  Jian Zhou,et al.  Where should the bugs be fixed? More accurate information retrieval-based bug localization based on bug reports , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[14]  LiGuo Huang,et al.  AutoODC: Automated generation of orthogonal defect classifications , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[15]  Xiangyu Zhang,et al.  Locating faults through automated predicate switching , 2006, ICSE.

[16]  Michael W. Godfrey,et al.  Automatic classication of large changes into maintenance categories , 2009, 2009 IEEE 17th International Conference on Program Comprehension.

[17]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[18]  W. Eric Wong,et al.  The DStar Method for Effective Software Fault Localization , 2014, IEEE Transactions on Reliability.

[19]  Jiawei Han,et al.  Generalized Fisher Score for Feature Selection , 2011, UAI.

[20]  Hong Cheng,et al.  Identifying bug signatures using discriminative graph mining , 2009, ISSTA.

[21]  David Lo,et al.  Extracting Paraphrases of Technical Terms from Noisy Parallel Software Corpora , 2009, ACL.

[22]  Howard J. Hamilton,et al.  Interestingness measures for data mining: A survey , 2006, CSUR.

[23]  Rajiv Gupta,et al.  Fault localization using value replacement , 2008, ISSTA '08.

[24]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[25]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[26]  Ferdian Thung,et al.  Automatic Defect Categorization , 2012, 2012 19th Working Conference on Reverse Engineering.

[27]  R. Suganya,et al.  Data Mining Concepts and Techniques , 2010 .

[28]  Baowen Xu,et al.  A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization , 2013, TSEM.

[29]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI.

[30]  Sunghun Kim,et al.  Predicting recurring crash stacks , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[31]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[32]  David Lo,et al.  Interactive fault localization leveraging simple user feedback , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[33]  Ken-ichi Matsumoto,et al.  Studying re-opened bugs in open source software , 2012, Empirical Software Engineering.

[34]  金田 重郎,et al.  C4.5: Programs for Machine Learning (書評) , 1995 .

[35]  David Lo,et al.  Extended comprehensive study of association measures for fault localization , 2014, J. Softw. Evol. Process..

[36]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[37]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[38]  Alessandro Orso,et al.  Are automated debugging techniques actually helping programmers? , 2011, ISSTA '11.

[39]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[40]  Andreas Zeller,et al.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT FSE.

[41]  Harald C. Gall,et al.  Cross-project defect prediction: a large scale experiment on data vs. domain vs. process , 2009, ESEC/SIGSOFT FSE.

[42]  Yann-Gaël Guéhéneuc,et al.  Support vector machines for anti-pattern detection , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[43]  James R. Larus,et al.  The use of program profiling for software maintenance with applications to the year 2000 problem , 1997, ESEC '97/FSE-5.

[44]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[45]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[46]  David Lo,et al.  Improved Duplicate Bug Report Identification , 2012, 2012 16th European Conference on Software Maintenance and Reengineering.

[47]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[48]  David Lo,et al.  Theory and Practice, Do They Match? A Case with Spectrum-Based Fault Localization , 2013, 2013 IEEE International Conference on Software Maintenance.

[49]  Siau-Cheng Khoo,et al.  A discriminative model approach for accurate duplicate bug report retrieval , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[50]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[51]  ZellerAndreas.  Isolating cause-effect chains from computer programs , 2002 .

[52]  Trishul M. Chilimbi,et al.  HOLMES: Effective statistical debugging via efficient path profiling , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[53]  Nathalie Japkowicz,et al.  The Class Imbalance Problem: Significance and Strategies , 2000 .

[54]  LiGuo Huang,et al.  AutoODC: Automated generation of Orthogonal Defect Classifications , 2011, ASE.

[55]  Gregory Tassey,et al.  Prepared for what , 2007 .

[56]  Bhavani M. Thuraisingham,et al.  Effective Software Fault Localization Using an RBF Neural Network , 2012, IEEE Transactions on Reliability.

[57]  R. Mahapatra,et al.  EFFECTIVE SOFTWARE FAULT LOCALIZATION USING GA-RBF NEURAL NETWORK , 2016 .

[58]  David G. Stork,et al.  Pattern Classification , 1973 .

[59]  Yi Zhang,et al.  Classifying Software Changes: Clean or Buggy? , 2008, IEEE Transactions on Software Engineering.

[60]  Gail C. Murphy,et al.  Who should fix this bug? , 2006, ICSE.

[61]  John G. Cleary,et al.  K*: An Instance-based Learner Using and Entropic Distance Measure , 1995, ICML.

[62]  David Lo,et al.  Diversity maximization speedup for fault localization , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[63]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[64]  Harald C. Gall,et al.  Cross-project Defect Prediction , 2009 .

[65]  Chao Liu,et al.  SOBER: statistical model-based bug localization , 2005, ESEC/FSE-13.

[66]  Gregg Rothermel,et al.  An empirical investigation of the relationship between spectra differences and regression faults , 2000 .