Metamorphic slice: An application in spectrum-based fault localization

Context: Because of its simplicity and effectiveness, Spectrum-Based Fault Localization (SBFL) has been one of the popular approaches towards fault localization. It utilizes the execution result of failure or pass, and the corresponding coverage information (such as program slice) to estimate the risk of being faulty for each program entity (such as statement). However, all existing SBFL techniques assume the existence of a test oracle to determine the execution result of a test case. But, it is common that test oracles do not exist, and hence the applicability of SBFL has been severely restricted. Objective: We aim at developing a framework that can extend the application of SBFL to the common situations where test oracles do not exist. Method: Our approach uses a new concept of metamorphic slice resulting from the integration of metamorphic testing and program slicing. In SBFL, instead of using the program slice and the result of failure or pass for an individual test case, a metamorphic slice and the result of violation or non-violation of a metamorphic relation are used. Since we need not know the execution result for an individual test case, the existence of a test oracle is no longer a requirement to apply SBFL. Results: An experimental study involving nine programs and three risk evaluation formulas was conducted. The results show that our proposed solution delivers a performance comparable to the performance observed by existing SBFL techniques for the situations where test oracles exist. Conclusion: With respect to the problem that SBFL is only applicable to programs with test oracles, we propose an innovative solution. Our solution is not only intuitively appealing and conceptually feasible, but also practically effective. Consequently, test oracles are no longer mandatory for SBFL, and hence the applicability of SBFL is significantly extended.

[1]  Tsong Yueh Chen,et al.  Case studies on the selection of useful relations in metamorphic testing , 2004 .

[2]  Mary Jean Harrold,et al.  Debugging in Parallel , 2007, ISSTA '07.

[3]  Baowen Xu,et al.  Testing and validating machine learning classifiers by metamorphic testing , 2011, J. Syst. Softw..

[4]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[5]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[6]  Tsong Yueh Chen,et al.  Metamorphic testing and beyond , 2003, Eleventh Annual International Workshop on Software Technology and Engineering Practice.

[7]  W. Chan,et al.  A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications , 2007, Int. J. Web Serv. Res..

[8]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[9]  Baowen Xu,et al.  Spectrum-Based Fault Localization: Testing Oracles are No Longer Mandatory , 2011, 2011 11th International Conference on Quality Software.

[10]  Baowen Xu,et al.  Application of Metamorphic Testing to Supervised Classifiers , 2009, 2009 Ninth International Conference on Quality Software.

[11]  Byoungju Choi,et al.  A family of code coverage-based heuristics for effective fault localization , 2010, J. Syst. Softw..

[12]  Chao Liu,et al.  SOBER: statistical model-based bug localization , 2005, ESEC/FSE-13.

[13]  Gregg Rothermel,et al.  An empirical investigation of the relationship between spectra differences and regression faults , 2000 .

[14]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[15]  Tsong Yueh Chen,et al.  An effective testing method for end-user programmers , 2005, WEUSE@ICSE.

[16]  Mark Harman,et al.  Pre/post conditioned slicing , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[17]  Armando Fox,et al.  Pinpoint: problem determination in large , 2002 .

[18]  Tsong Yueh Chen,et al.  Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging , 2011, IEEE Transactions on Software Engineering.

[19]  Mark Harman,et al.  A survey of empirical results on program slicing , 2004, Adv. Comput..

[20]  Wing Hung Wong,et al.  SeqMap: mapping massive amount of oligonucleotides to the genome , 2008, Bioinform..

[21]  T. H. Tse,et al.  An empirical comparison between direct and indirect test result checking approaches , 2006, SOQUA '06.

[22]  Yu Qi,et al.  Smart debugging software architectural design in SDL , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[23]  Lei Zhao,et al.  A Crosstab-based Statistical Method for Effective Fault Localization , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[24]  W. K. Chan,et al.  Experimental study to compare the use of metamorphic testing and assertion checking , 2009 .

[25]  Huai Liu,et al.  An innovative approach for testing bioinformatics programs using metamorphic testing , 2009, BMC Bioinformatics.

[26]  Gregory W. Corder,et al.  Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach , 2009 .

[27]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[28]  Peter Zoeteweij,et al.  An Evaluation of Similarity Coefficients for Software Fault Localization , 2006, 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06).

[29]  Scott N. Woodfield,et al.  Evaluating the effectiveness of reliability-assurance techniques , 1989, J. Syst. Softw..

[30]  William G. Griswold,et al.  SIGSOFT 2002/FSE 10 : proceedings of the tenth ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-10), November 18-22, 2002, Charleston, South Carolina, USA , 2002 .

[31]  Ali Selamat,et al.  Information and Software Technology , 2014 .

[32]  Mark Harman,et al.  Conditioned slicing supports partition testing , 2002, Softw. Test. Verification Reliab..

[33]  Mary Jean Harrold,et al.  Empirical evaluation of the tarantula automatic fault-localization technique , 2005, ASE.

[34]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[35]  Chao Liu,et al.  Statistical Debugging: A Hypothesis Testing-Based Approach , 2006, IEEE Transactions on Software Engineering.

[36]  Tsong Yueh Chen,et al.  An effective testing method for end-user programmers , 2005, ACM SIGSOFT Softw. Eng. Notes.

[37]  James R. Larus,et al.  The use of program profiling for software maintenance with applications to the year 2000 problem , 1997, ESEC '97/FSE-5.

[38]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..