Spectrum-Based Fault Localization: Testing Oracles are No Longer Mandatory

Spectrum-based Fault Localization (SBFL) is one of the most popular approaches for locating software faults, and has received much attention because of its simplicity and effectiveness. It utilizes the execution result of each test case (failure or pass) and the corresponding coverage information to evaluate the likelihood of each program entity (e.g., a statement or a predicate) being faulty. Different formulas for computing such likelihood have been proposed based on different intuitions. All existing SBFL techniques have assumed the existence of a testing oracle, that is, a mechanism which can determine whether the execution of a test case fails or passes. However, such an assumption does not always hold. Recently, metamorphic testing has been proposed to alleviate the oracle problem. Thus, it is a natural extension to investigate how it can help SBFL techniques to locate faults even without using a testing oracle. Based on the framework of metamorphic testing, we have developed a novel concept of mice as a counterpart of the slice used in the current SBFL techniques. More precisely, in the absence of a testing oracle, we can determine whether an expected characterization of the program is satisfied. The outcomes of dissatisfaction or satisfaction of an expected characterization are then regarded as the counterparts of failed or passed executions, respectively, when a testing oracle exists. Since our approach does not require the existence of a testing oracle, it significantly enhances the applicability of SBFL techniques. Case studies on three popular SBFL techniques (Tarantula, Ochiai and Jaccard) with 9 applications are reported to demonstrate the use of the proposed fault localization technique.

[1]  Gail E. Kaiser,et al.  Automatic system testing of programs without test oracles , 2009, ISSTA.

[2]  Mark Harman,et al.  A survey of empirical results on program slicing , 2004, Adv. Comput..

[3]  Chao Liu,et al.  SOBER: statistical model-based bug localization , 2005, ESEC/FSE-13.

[4]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[5]  Gregg Rothermel,et al.  An empirical investigation of the relationship between spectra differences and regression faults , 2000 .

[6]  Alex Aiken,et al.  Cooperative Bug Isolation , 2007 .

[7]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[8]  Baowen Xu,et al.  Application of Metamorphic Testing to Supervised Classifiers , 2009, 2009 Ninth International Conference on Quality Software.

[9]  Peter Zoeteweij,et al.  An Evaluation of Similarity Coefficients for Software Fault Localization , 2006, 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06).

[10]  Michael A. Charleston,et al.  Differential variability analysis of gene expression and its application to human diseases , 2008, ISMB.

[11]  Janet M. Thornton,et al.  Software engineering challenges in bioinformatics , 2004, Proceedings. 26th International Conference on Software Engineering.

[12]  James R. Larus,et al.  The use of program profiling for software maintenance with applications to the year 2000 problem , 1997, ESEC '97/FSE-5.

[13]  Gail E. Kaiser,et al.  Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles , 2009, 2009 International Conference on Software Testing Verification and Validation.

[14]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[15]  Hsin Pan,et al.  Heuristics for Automatic Localization of Software Faults , 1992 .

[16]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[17]  Arnaud Gotlieb,et al.  Automated metamorphic testing , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[18]  Yu Qi,et al.  Smart debugging software architectural design in SDL , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[19]  Scott N. Woodfield,et al.  Evaluating the effectiveness of reliability-assurance techniques , 1989, J. Syst. Softw..

[20]  Huai Liu,et al.  An innovative approach for testing bioinformatics programs using metamorphic testing , 2009, BMC Bioinformatics.

[21]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[22]  Mary Jean Harrold,et al.  Debugging in Parallel , 2007, ISSTA '07.

[23]  C. D. dos Remedios,et al.  Customising an antibody leukocyte capture microarray for systemic lupus erythematosus: Beyond biomarker discovery , 2010, Proteomics. Clinical applications.

[24]  Chao Liu,et al.  Statistical Debugging: A Hypothesis Testing-Based Approach , 2006, IEEE Transactions on Software Engineering.

[25]  Tsong Yueh Chen,et al.  Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging , 2011, IEEE Transactions on Software Engineering.

[26]  Mark Harman,et al.  An empirical study of static program slice size , 2007, TSEM.

[27]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[28]  Wing Hung Wong,et al.  SeqMap: mapping massive amount of oligonucleotides to the genome , 2008, Bioinform..

[29]  Lei Zhao,et al.  A Crosstab-based Statistical Method for Effective Fault Localization , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[30]  Gail E. Kaiser,et al.  Metamorphic testing techniques to detect defects in applications without test oracles , 2010 .

[31]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[32]  Byoungju Choi,et al.  A family of code coverage-based heuristics for effective fault localization , 2010, J. Syst. Softw..

[33]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[34]  W. Chan,et al.  A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications , 2007, Int. J. Web Serv. Res..

[35]  Mark Harman,et al.  Theoretical foundations of dynamic program slicing , 2006, Theor. Comput. Sci..