A consensus‐based strategy to improve the quality of fault localization

A vast number of software fault localization techniques have been proposed recently with the growing realization that manual debugging is time‐consuming, tedious, and error‐prone, and fault localization is one of the most expensive debugging activities. Although some techniques perform better than others on a large number of data sets, they do not do so on all data sets and therefore, the actual quality of fault localization can vary considerably by using just one technique. This paper proposes the use of a consensus‐based strategy that combines the results of multiple fault localization techniques to consistently provide high quality performance, irrespective of data set. Empirical evidence based on case studies conducted on six sets of programs (seven programs of the Siemens suite, and the gzip, grep, make, space, and Ant programs) and three different fault localization techniques (Tarantula, Ochiai, and H3) suggests that the consensus‐based strategy holds merit and generally provides close to the best, if not the best, results. Empirically, we show that this is true of both single‐fault and multifault programs. Additionally, the consensus‐based strategy makes use of techniques that all operate on the same set of input data, minimizing the overhead. It is also simple to include or exclude techniques from consensus, making it an easily extensible or tractable strategy. Copyright © 2011 John Wiley & Sons, Ltd.

[1]  Iris Vessey,et al.  Expertise in Debugging Computer Programs , 1984 .

[2]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[3]  Gholamreza Latif Shabgahi,et al.  A fuzzy voting scheme for hardware and software fault tolerant systems , 2005, Fuzzy Sets Syst..

[4]  Liming Chen,et al.  N-VERSION PROGRAMMINC: A FAULT-TOLERANCE APPROACH TO RELlABlLlTY OF SOFTWARE OPERATlON , 1995, Twenty-Fifth International Symposium on Fault-Tolerant Computing, 1995, ' Highlights from Twenty-Five Years'..

[5]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[6]  Xiangyu Zhang,et al.  Locating faults through automated predicate switching , 2006, ICSE.

[7]  Eugene H. Spafford,et al.  Debugging with dynamic slicing and backtracking , 1993, Softw. Pract. Exp..

[8]  Peter Zoeteweij,et al.  A practical evaluation of spectrum-based fault localization , 2009, J. Syst. Softw..

[9]  C. Dwork,et al.  Rank Aggregation Revisited , 2002 .

[10]  P. Jaccard.  THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1 , 1912 .

[11]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[12]  K. Arrow.  A Difficulty in the Concept of Social Welfare , 1950, Journal of Political Economy.

[13]  Atri Rudra,et al.  Ordering by weighted number of wins gives a good ranking for weighted tournaments , 2006, SODA '06.

[14]  T. H. Tse,et al.  Capturing propagation of infected program states , 2009, ESEC/FSE '09.

[15]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[16]  Iris Vessey,et al.  Expertise in Debugging Computer Programs: A Process Analysis , 1984, Int. J. Man Mach. Stud..

[17]  H. Cleve,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[18]  Lei Zhao,et al.  A Crosstab-based Statistical Method for Effective Fault Localization , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[19]  Mary Jean Harrold,et al.  Debugging in Parallel , 2007, ISSTA '07.

[20]  Chao Liu,et al.  Statistical Debugging: A Hypothesis Testing-Based Approach , 2006, IEEE Transactions on Software Engineering.

[21]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[22]  J. von Neumann,et al.  Probabilistic Logic and the Synthesis of Reliable Organisms from Unreliable Components , 1956 .

[23]  Kai-Yuan Cai,et al.  Effective Fault Localization using Code Coverage , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[24]  Joseph Robert Horgan,et al.  Fault localization using execution slices and dataflow tests , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[25]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[26]  Nancy G. Leveson,et al.  An experimental evaluation of the assumption of independence in multiversion programming , 1986, IEEE Transactions on Software Engineering.

[27]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[28]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[29]  Mark Weiser,et al.  Programmers use slices when debugging , 1982, CACM.

[30]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[31]  Yu Qi,et al.  Effective program debugging based on execution slices and inter-block data dependency , 2006, J. Syst. Softw..

[32]  Nir Ailon,et al.  Aggregating inconsistent information: Ranking and clustering , 2008 .

[33]  L. Cranor,et al.  Declared-strategy voting: an instrument for group decision-making , 1996 .

[34]  Moni Naor,et al.  Rank aggregation methods for the Web , 2001, WWW '01.

[35]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[36]  W. Eric Wong,et al.  On the Consensus-Based Application of Fault Localization Techniques , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops.

[37]  Vincent Conitzer,et al.  Improved Bounds for Computing Kemeny Rankings , 2006, AAAI.

[38]  Yan Shi,et al.  Using an RBF Neural Network to Locate Program Bugs , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).