Spectrum-based Software Fault Localization: A Survey of Techniques, Advances, and Challenges

Despite being one of the most basic tasks in software development, debugging is still performed in a mostly manual way, leading to high cost and low performance. To address this problem, researchers have studied promising approaches, such as Spectrum-based Fault Localization (SFL) techniques, which pinpoint program elements more likely to contain faults. This survey discusses the state-of-the-art of SFL, including the different techniques that have been proposed, the type and number of faults they address, the types of spectra they use, the programs they utilize in their validation, the testing data that support them, and their use at industrial settings. Notwithstanding the advances, there are still challenges for the industry to adopt these techniques, which we analyze in this paper. SFL techniques should propose new ways to generate reduced sets of suspicious entities, combine different spectra to fine-tune the fault localization ability, use strategies to collect fine-grained coverage levels from suspicious coarser levels for balancing execution costs and output precision, and propose new techniques to cope with multiple-fault programs. Moreover, additional user studies are needed to understand better how SFL techniques can be used in practice. We conclude by presenting a concept map about topics and challenges for future research in SFL.

[1]  Peter Fritzson,et al.  Generalized algorithmic debugging and testing , 1991, LOPL.

[2]  Alessandro Orso,et al.  MintHint: automated synthesis of repair hints , 2013, ICSE.

[3]  Eunseok Lee,et al.  Empirical evaluation of existing algorithms of spectrum based fault localization , 2014, The International Conference on Information Networking 2014 (ICOIN2014).

[4]  Alberto Sánchez Automatic Error Detection Techniques Based on Dynamic Invariants THESIS submitted in partial fulfillment of the requirements for the degree of MASTER OF , 2007 .

[5]  Baowen Xu,et al.  Revisit of Automatic Debugging via Human Focus-Tracking Analysis , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[6]  Andreas Zeller,et al.  Lightweight Defect Localization for Java , 2005, ECOOP.

[7]  T. H. Tse,et al.  Non-parametric statistical fault localization , 2011, J. Syst. Softw..

[8]  Xiangyu Zhang,et al.  Experimental evaluation of using dynamic slices for fault location , 2005, AADEBUG'05.

[9]  Kai-Yuan Cai,et al.  Does the Failing Test Execute a Single or Multiple Faults? An Approach to Classifying Failing Tests , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[10]  Shounak Roychowdhury Ensemble of feature selectors for software fault localization , 2012, 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[11]  Lingfeng Bao,et al.  “Automated Debugging Considered Harmful” Considered Harmful: A User Study Revisiting the Usefulness of Spectra-Based Fault Localization Techniques with Professionals Using Real Bugs from Large Systems , 2016, 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[12]  Ehud Shapiro,et al.  Algorithmic Program Debugging , 1983 .

[13]  Martin Burger,et al.  Minimizing reproduction of software failures , 2011, ISSTA '11.

[14]  James H. Andrews,et al.  Evaluating the Accuracy of Fault Localization Techniques , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[15]  Dianxiang Xu,et al.  Towards Better Fault Localization: A Crosstab-Based Statistical Approach , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[16]  Shin Yoo,et al.  Ask the Mutants: Mutating Faulty Programs for Fault Localization , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[17]  Alessandro Orso,et al.  Isolating failure causes through test case generation , 2012, ISSTA 2012.

[18]  James A. Jones,et al.  Fault density, fault types, and spectra-based fault localization , 2015, Empirical Software Engineering.

[19]  Ji Wu,et al.  Finding suspicious patterns of object-oriented programs based on variance analysis , 2010, 2010 Seventh International Conference on Fuzzy Systems and Knowledge Discovery.

[20]  W. Eric Wong,et al.  Insights on Fault Interference for Programs with Multiple Bugs , 2009, 2009 20th International Symposium on Software Reliability Engineering.

[21]  Rui Abreu,et al.  A Topology-Based Model for Estimating the Diagnostic Efficiency of Statistics-Based Approaches , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[22]  Jian Liu,et al.  Program structure aware fault localization , 2014, InnoSWDev@SIGSOFT FSE.

[23]  Rui Abreu,et al.  Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators , 2013, ISSTA.

[24]  Serge Demeyer,et al.  Localising faults in test execution traces , 2015, IWPSE.

[25]  Elliot Soloway,et al.  PROUST: Knowledge-Based Program Understanding , 1984, IEEE Transactions on Software Engineering.

[26]  Mary Jean Harrold,et al.  Debugging in Parallel , 2007, ISSTA '07.

[27]  David Lo,et al.  Extended comprehensive study of association measures for fault localization , 2014, J. Softw. Evol. Process..

[28]  Yves Le Traon,et al.  Metallaxis‐FL: mutation‐based fault localization , 2015, Softw. Test. Verification Reliab..

[29]  Franz Wotawa,et al.  Does Testing Help to Reduce the Number of Potentially Faulty Statements in Debugging? , 2010, TAIC PART.

[30]  Alessandro Orso,et al.  Are automated debugging techniques actually helping programmers? , 2011, ISSTA '11.

[31]  Peggy Aldrich Kidwell Stalking the Elusive Computer Bug , 1998, IEEE Ann. Hist. Comput..

[32]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[33]  Claes Wohlin,et al.  Systematic literature studies: Database searches vs. backward snowballing , 2012, Proceedings of the 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement.

[34]  James S. Collofello,et al.  Towards automatic software fault location through decision-to-decision path analysis , 1899 .

[35]  Marcos Lordello Chaim,et al.  Data-Flow Testing in the Large , 2014, ICST.

[36]  Trishul M. Chilimbi,et al.  HOLMES: Effective statistical debugging via efficient path profiling , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[37]  Baowen Xu,et al.  Isolating Suspiciousness from Spectrum-Based Fault Localization Techniques , 2010, 2010 10th International Conference on Quality Software.

[38]  James A. Jones,et al.  Semantic fault diagnosis: automatic natural-language fault descriptions , 2012, SIGSOFT FSE.

[39]  T. H. Tse,et al.  Is non-parametric hypothesis testing model robust for statistical fault localization? , 2009, Inf. Softw. Technol..

[40]  Pragya Agarwal,et al.  Fault-localization techniques for software systems: a literature review , 2014, SOEN.

[41]  Xiangyu Zhang,et al.  Locating faults using multiple spectra-specific models , 2011, SAC.

[42]  Alessandro Orso,et al.  Rapid: Identifying Bug Signatures to Support Debugging Activities , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[43]  Marcelo d'Amorim,et al.  Prevalence of Single-Fault Fixes and Its Impact on Fault Localization , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[44]  Wes Masri,et al.  Prevalence of coincidental correctness and mitigation of its impact on fault localization , 2014, TSEM.

[45]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[46]  Sarfraz Khurshid,et al.  Software fault localization using feature selection , 2011, MALETS '11.

[47]  Robert Hirschfeld,et al.  Follow the path: Debugging state anomalies along execution histories , 2014, 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE).

[48]  Akbar Siami Namin,et al.  Trimming Test Suites with Coincidentally Correct Test Cases for Enhancing Fault Localizations , 2014, 2014 IEEE 38th Annual Computer Software and Applications Conference.

[49]  Rajiv Gupta,et al.  Fault localization using value replacement , 2008, ISSTA '08.

[50]  Sarfraz Khurshid,et al.  Improving bug localization using structured information retrieval , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[51]  Andreas Zeller,et al.  Where is the bug and how is it fixed? an experiment with practitioners , 2017, ESEC/SIGSOFT FSE.

[52]  Thomas Zimmermann,et al.  Extraction of bug localization benchmarks from history , 2007, ASE.

[53]  Lei Zhao,et al.  PAFL: Fault Localization via Noise Reduction on Coverage Vector , 2011, SEKE.

[54]  Joseph Robert Horgan,et al.  Fault localization using execution slices and dataflow tests , 1995, Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95.

[55]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[56]  Michael D. Ernst,et al.  Evaluating and Improving Fault Localization , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[57]  Alessandro Orso,et al.  F3: fault localization for field failures , 2013, ISSTA.

[58]  Yu Qi,et al.  Effective program debugging based on execution slices and inter-block data dependency , 2006, J. Syst. Softw..

[59]  Henrique Madeira,et al.  Emulation of Software Faults: A Field Data Study and a Practical Approach , 2006, IEEE Transactions on Software Engineering.

[60]  Shin Yoo,et al.  Evolving Human Competitive Spectra-Based Fault Localisation Techniques , 2012, SSBSE.

[61]  David Lo,et al.  Theory and Practice, Do They Match? A Case with Spectrum-Based Fault Localization , 2013, 2013 IEEE International Conference on Software Maintenance.

[62]  Byoungju Choi,et al.  A family of code coverage-based heuristics for effective fault localization , 2010, J. Syst. Softw..

[63]  Chao Liu,et al.  A Systematic Study of Failure Proximity , 2008, IEEE Transactions on Software Engineering.

[64]  Chao Liu,et al.  Mining Behavior Graphs for "Backtrace" of Noncrashing Bugs , 2005, SDM.

[65]  Baowen Xu,et al.  An In-Depth Study of the Efficiency of Risk Evaluation Formulas for Multi-Fault Localization , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C).

[66]  Lei Zhao,et al.  Context-Aware Fault Localization via Control Flow Analysis , 2011, J. Softw..

[67]  Joseph Robert Horgan,et al.  Dynamic program slicing , 1990, PLDI '90.

[68]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[69]  Friedrich Steimann,et al.  More Debugging in Parallel , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[70]  Latifur Khan,et al.  Software Fault Localization Using N-gram Analysis , 2008, WASA.

[71]  Lee Naish,et al.  A model for spectra-based software diagnosis , 2011, TSEM.

[72]  Zhendong Su,et al.  Context-aware statistical debugging: from bug predictors to faulty control flow paths , 2007, ASE.

[73]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[74]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[75]  Bixin Li,et al.  Program slicing spectrum-based software fault localization , 2011, SEKE.

[76]  David Lo,et al.  Should I follow this fault localization tool’s output? , 2014, Empirical Software Engineering.

[77]  Xiaoguang Mao,et al.  Fault localization via behavioral models , 2012, 2012 IEEE International Conference on Computer Science and Automation Engineering.

[78]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[79]  Mohammad Amin Alipour Automated Fault Localization Techniques ; A Survey , 2013 .

[80]  Michael I. Jordan,et al.  Statistical debugging: simultaneous identification of multiple bugs , 2006, ICML '06.

[81]  Robert Balzer,et al.  EXDAMS: extendable debugging and monitoring system , 1969, AFIPS '69 (Spring).

[82]  Shin Yoo,et al.  FLUCCS: using code and change metrics to improve fault localization , 2017, ISSTA.

[83]  Serge Demeyer,et al.  Fine-tuning spectrum based fault localisation with frequent method item sets , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[84]  Lee Naish,et al.  Spectral Debugging with Weights and Incremental Ranking , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[85]  Liang Guo,et al.  Accurately Choosing Execution Runs for Software Fault Localization , 2006, CC.

[86]  Markus Stumptner,et al.  Model-Based Debugging or How to Diagnose Programs Automatically , 2002, IEA/AIE.

[87]  Alberto J. Cañas,et al.  A TEORIA SUBJACENTE AOS MAPAS CONCEITUAIS E COMO ELABORÁ-LOS E USÁ-LOS * THE THEORY UNDERLYING CONCEPT MAPS AND HOW TO CONSTRUCT AND USE THEM , 2010 .

[88]  Yves Le Traon,et al.  Improving test suites for efficient fault localization , 2006, ICSE.

[89]  T. H. Tse,et al.  Capturing propagation of infected program states , 2009, ESEC/FSE '09.

[90]  Rong Chen,et al.  Probabilistic reasoning in diagnosing causes of program failures , 2016, Softw. Test. Verification Reliab..

[91]  Ferdian Thung,et al.  Automatic Defect Categorization , 2012, 2012 19th Working Conference on Reverse Engineering.

[92]  Baowen Xu,et al.  A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization , 2013, TSEM.

[93]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[94]  W. Eric Wong,et al.  On the equivalence of certain fault localization techniques , 2011, SAC '11.

[95]  Mechelle Gittens,et al.  Discovering  the Fault Origin from Field Traces , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[96]  Xiaofeng Xu,et al.  Ties within Fault Localization rankings: Exposing and Addressing the Problem , 2011, Int. J. Softw. Eng. Knowl. Eng..

[97]  David Lo,et al.  Are faults localizable? , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[98]  Lee Naish,et al.  Statements versus Predicates in Spectral Bug Localization , 2010, 2010 Asia Pacific Software Engineering Conference.

[99]  Brian A. Malloy,et al.  A Linear Programming Approach for Automated Localization of Multiple Faults , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[100]  Andreas Zeller,et al.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT FSE.

[101]  Lee Naish,et al.  Duals in Spectral Fault Localization , 2013, 2013 22nd Australian Software Engineering Conference.

[102]  Lin Chen,et al.  Wielding Statistical Fault Localization Statistically , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[103]  Jason Lee,et al.  Improving spectral‐based fault localization using static analysis , 2017, Softw. Pract. Exp..

[104]  Xiang Chen,et al.  Fault Localization Based on Multi-level Similarity of Execution Traces , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[105]  George Nagy,et al.  A Step Toward Automatic Analysis of Student Programming Errors in a Batch Environment , 1974, Int. J. Man Mach. Stud..

[106]  James A. Jones,et al.  Software Behavior and Failure Clustering: An Empirical Study of Fault Causality , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[107]  H. Agrawal,et al.  Bibliography on debugging and backtracking , 1989, SOEN.

[108]  Shin Hong,et al.  MUSEUM: Debugging real-world multilingual programs using mutation analysis , 2017, Inf. Softw. Technol..

[109]  Jeffrey S. Foster,et al.  A comparison of bug finding tools for Java , 2004, 15th International Symposium on Software Reliability Engineering.

[110]  Tao Zhang,et al.  Uniformly Evaluating and Comparing Ranking Metrics for Spectral Fault Localization , 2014, 2014 14th International Conference on Quality Software.

[111]  Sudipto Ghosh,et al.  Tester Feedback Driven Fault Localization , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[112]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[113]  Xiangyu Zhang,et al.  Towards locating execution omission errors , 2007, PLDI '07.

[114]  Andy Podgurski,et al.  Causal inference for statistical fault localization , 2010, ISSTA '10.

[115]  Atif M. Memon,et al.  Disqover: Debugging via code sequence covers , 2015, 2015 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[116]  W. Eric Wong,et al.  On the Consensus-Based Application of Fault Localization Techniques , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops.

[117]  Xiaofeng Xu,et al.  A Grouping-Based Strategy to Improve the Effectiveness of Fault Localization Techniques , 2010, 2010 10th International Conference on Quality Software.

[118]  James A. Jones,et al.  Concept-based failure clustering , 2012, SIGSOFT FSE.

[119]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[120]  Marcelo d'Amorim,et al.  Fault-localization using dynamic slicing and change impact analysis , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[121]  Wang Yu,et al.  State dependency probabilistic model for fault localization , 2015, Inf. Softw. Technol..

[122]  Xinrui Guo,et al.  First, Debug the Test Oracle , 2015, IEEE Transactions on Software Engineering.

[123]  W. Eric Wong,et al.  Software Fault Localization Using DStar (D*) , 2012, 2012 IEEE Sixth International Conference on Software Security and Reliability.

[124]  Wang Tiantian,et al.  Comprehension oriented software fault location , 2011, Proceedings of 2011 International Conference on Computer Science and Network Technology.

[125]  Friedrich Steimann,et al.  A Simple Coverage-Based Locator for Multiple Faults , 2009, 2009 International Conference on Software Testing Verification and Validation.

[126]  Tao Xie,et al.  Interactive Fault Localization Using Test Information , 2009, Journal of Computer Science and Technology.

[127]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[128]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[129]  Lei Zhao,et al.  Execution-Aware Fault Localization Based on the Control Flow Analysis , 2010, ICICA.

[130]  Marcos Lordello Chaim,et al.  An efficient bitwise algorithm for intra-procedural data-flow testing coverage , 2013, Inf. Process. Lett..

[131]  Sai Zhang,et al.  Software bug localization with markov logic , 2014, ICSE Companion.

[132]  Peter Zoeteweij,et al.  Simultaneous debugging of software faults , 2011, J. Syst. Softw..

[133]  Tao Wang,et al.  Automated path generation for software fault localization , 2005, ASE '05.

[134]  Phyllis G. Frankl,et al.  Empirical evaluation of the textual differencing regression testing technique , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[135]  Bogdan Korel,et al.  PELAS - Program Error-Locating Assistant System , 1988, IEEE Trans. Software Eng..

[136]  David Lo,et al.  Information retrieval and spectrum based bug localization: better together , 2015, ESEC/SIGSOFT FSE.

[137]  Thomas G. Evans,et al.  On-line debugging techniques: a survey , 1966, AFIPS '66 (Fall).

[138]  David Lo,et al.  Practitioners' expectations on automated fault localization , 2016, ISSTA.

[139]  Lars Grunske,et al.  A learning-to-rank based fault localization approach using likely invariants , 2016, ISSTA.

[140]  Gregory Tassey,et al.  Prepared for what , 2007 .

[141]  Akbar Siami Namin,et al.  How Significant is the Effect of Fault Interactions on Coverage-Based Fault Localizations? , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[142]  Bhavani M. Thuraisingham,et al.  Effective Software Fault Localization Using an RBF Neural Network , 2012, IEEE Transactions on Reliability.

[143]  Aladdin: Assembly Language Assertion Driven Debugging Interpreter , 1979, IEEE Transactions on Software Engineering.

[144]  Tao Xie,et al.  Test input reduction for result inspection to facilitate fault localization , 2010, Automated Software Engineering.

[145]  Kai-Yuan Cai,et al.  Using Partition Information to Prioritize Test Cases for Fault Localization , 2015, 2015 IEEE 39th Annual Computer Software and Applications Conference.

[146]  Rui Abreu,et al.  Cues for scent intensification in debugging , 2013, 2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[147]  Xing Xu,et al.  A New Spectrum-based Fault Localization Method by Using Clustering Algorithm , 2012 .

[148]  Shujuan Jiang,et al.  HSFal: Effective fault localization using hybrid spectrum of full slices and execution slices , 2014, J. Syst. Softw..

[149]  Xiaoyan Wang,et al.  Automated fault localization via hierarchical multiple predicate switching , 2015, J. Syst. Softw..

[150]  Jian Xu,et al.  A general noise-reduction framework for fault localization of Java programs , 2013, Inf. Softw. Technol..

[151]  Martin Monperrus,et al.  Test case purification for improving fault localization , 2014, SIGSOFT FSE.

[152]  Akbar Siami Namin,et al.  Measuring the Odds of Statements Being Faulty , 2013, Ada-Europe.

[153]  Chao Liu,et al.  Statistical Debugging: A Hypothesis Testing-Based Approach , 2006, IEEE Transactions on Software Engineering.

[154]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[155]  Shin Hong,et al.  Mutation-Based Fault Localization for Real-World Multilingual Programs (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[156]  Ting Chen,et al.  Statistical debugging using compound boolean predicates , 2007, ISSTA '07.

[157]  Hong Cheng,et al.  Identifying bug signatures using discriminative graph mining , 2009, ISSTA.

[158]  Xia Li,et al.  Boosting spectrum-based fault localization using PageRank , 2017, ISSTA.

[159]  Jean-Pierre H. Laurent,et al.  LAURA, A System to Debug Student Programs , 1980, Artif. Intell..

[160]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[161]  Sudipto Ghosh,et al.  Proximity based weighting of test cases to improve spectrum based fault localization , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[162]  Higor Amario de Souza,et al.  Adding context to fault localization with integration coverage , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[163]  Peter Zoeteweij,et al.  Localizing Software Faults Simultaneously , 2009, 2009 Ninth International Conference on Quality Software.

[164]  Lars Asplund Reliable Software Technologies — Ada-Europe , 1998, Lecture Notes in Computer Science.

[165]  W. Eric Wong,et al.  Combining mutation and fault localization for automated program debugging , 2014, J. Syst. Softw..

[166]  Lei Zhao,et al.  Regression Identification of Coincidental Correctness via Weighted Clustering , 2015, 2015 IEEE 39th Annual Computer Software and Applications Conference.

[167]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[168]  Westley Weimer,et al.  Patches as better bug reports , 2006, GPCE '06.

[169]  Xiangyu Zhang,et al.  Locating faults through automated predicate switching , 2006, ICSE.

[170]  Lars Grunske,et al.  A Critical Evaluation of Spectrum-Based Fault Localization Techniques on a Large-Scale Software System , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[171]  Jian Zhang,et al.  A theoretical analysis on cloning the failed test cases to improve spectrum-based fault localization , 2017, J. Syst. Softw..

[172]  Peter Zoeteweij,et al.  A practical evaluation of spectrum-based fault localization , 2009, J. Syst. Softw..

[173]  Chin-Yu Huang,et al.  A Study of Modified Testing-Based Fault Localization Method , 2008, 2008 14th IEEE Pacific Rim International Symposium on Dependable Computing.

[174]  Jian Liu,et al.  A Synergistic Analysis Method for Explaining Failed Regression Tests , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[175]  James R. Larus,et al.  The use of program profiling for software maintenance with applications to the year 2000 problem , 1997, ESEC '97/FSE-5.

[176]  Leonardo Mariani,et al.  Dynamic Analysis for Diagnosing Integration Faults , 2011, IEEE Transactions on Software Engineering.