Prioritizing tests for fault localization through ambiguity group reduction

In practically all development processes, regression tests are used to detect the presence of faults after a modification. If faults are detected, a fault localization algorithm can be used to reduce the manual inspection cost. However, while using test case prioritization to enhance the rate of fault detection of the test suite (e.g., statement coverage), the diagnostic information gain per test is not optimal, which results in needless inspection cost during diagnosis. We present RAPTOR, a test prioritization algorithm for fault localization, based on reducing the similarity between statement execution patterns as the testing progresses. Unlike previous diagnostic prioritization algorithms, RAPTOR does not require false negative information, and is much less complex. Experimental results from the Software Infrastructure Repository's benchmarks show that RAPTOR is the best technique under realistic conditions, with average cost reductions of 40% with respect to the next best technique, with negligible impact on fault detection capability.

[1]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[2]  Gregory M. Kapfhammer,et al.  An empirical study of incorporating cost into test suite reduction and prioritization , 2009, SAC '09.

[3]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[4]  Rui Abreu,et al.  Prioritizing tests for software fault diagnosis , 2011, Softw. Pract. Exp..

[5]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[6]  John T. Stasko,et al.  Visualization of test information to assist fault localization , 2002, ICSE '02.

[7]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[8]  Krishna R. Pattipati,et al.  Sequential testing algorithms for multiple fault diagnosis , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[9]  Rui Abreu,et al.  Spectrum-Based Sequential Diagnosis , 2011, AAAI.

[10]  Raúl A. Santelices,et al.  Lightweight fault-localization using multiple coverage types , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[11]  Tsong Yueh Chen,et al.  How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[12]  Peter Zoeteweij,et al.  An observation-based model for fault localization , 2008, WODA.

[13]  Ben Liblit.  Cooperative debugging with five hundred million test cases , 2008, ISSTA '08.

[14]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[15]  Chao Liu,et al.  SOBER: statistical model-based bug localization , 2005, ESEC/FSE-13.

[16]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[17]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[18]  Peter Zoeteweij,et al.  A practical evaluation of spectrum-based fault localization , 2009, J. Syst. Softw..

[19]  Brian C. Williams,et al.  Diagnosing Multiple Faults , 1987, Artif. Intell..

[20]  Arjan J. C. van Gemund,et al.  Prioritizing Tests for Software Fault Localization , 2010, 2010 10th International Conference on Quality Software.

[21]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[22]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[23]  Gregg Rothermel,et al.  Selecting a Cost-Effective Test Case Prioritization Technique , 2004, Software Quality Journal.

[24]  Arjan J. C. van Gemund,et al.  Modeling the Diagnostic Efficiency of Regression Test Suites , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[25]  Jeffrey M. Voas,et al.  PIE: A Dynamic Failure-Based Technique , 1992, IEEE Trans. Software Eng..

[26]  Mary Jean Harrold,et al.  Recomputing Coverage Information to Assist Regression Testing , 2009, IEEE Transactions on Software Engineering.

[27]  Yi Zhang,et al.  Classifying Software Changes: Clean or Buggy? , 2008, IEEE Transactions on Software Engineering.

[28]  Rui Abreu,et al.  An empirical study on the usage of testability information to fault localization in software , 2011, SAC '11.

[29]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[30]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[31]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[32]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[33]  Lei Zhao,et al.  A Crosstab-based Statistical Method for Effective Fault Localization , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[34]  Sheng Ma,et al.  Intelligent probing: A cost-effective approach to fault diagnosis in computer networks , 2002, IBM Syst. J..

[35]  Rui Abreu,et al.  Zoltar: A Toolset for Automatic Fault Localization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[36]  Shing-Chi Cheung,et al.  Taming coincidental correctness: Coverage refinement with context patterns to improve fault localization , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[37]  Andy Podgurski,et al.  The Probabilistic Program Dependence Graph and Its Application to Fault Diagnosis , 2008, IEEE Transactions on Software Engineering.

[38]  Lu Zhang,et al.  Eliminating harmful redundancy for testing-based fault localization using test suite reduction: an experimental study , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[39]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[40]  Peter Zoeteweij,et al.  Spectrum-Based Multiple Fault Localization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[41]  A.J.C. van Gemund,et al.  On the Accuracy of Spectrum-based Fault Localization , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[42]  Krishna R. Pattipati,et al.  Test sequencing algorithms with unreliable tests , 1999, IEEE Trans. Syst. Man Cybern. Part A.

[43]  Yves Le Traon,et al.  Improving test suites for efficient fault localization , 2006, ICSE.

[44]  Markus Stumptner,et al.  Evaluating Models for Model-Based Debugging , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[45]  G. Stenbakken,et al.  Ambiguity groups and testability , 1989 .

[46]  Steven P. Reiss,et al.  Fault localization with nearest neighbor queries , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..