Prioritizing test cases for regression testing

Test case prioritization techniques schedule test cases in an order that increases their effectiveness in meeting some performance goal. One performance goal, rate of fault detection, is a measure of how quickly faults are detected within the testing process; an improved rate of fault detection can provide faster feedback on the system under test, and let software engineers begin locating and correcting faults earlier than might otherwise be possible. In previous work, we reported the results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: (1) can prioritization techniques be effective when aimed at specific modified versions; (2) what tradeoffs exist between fine granularity and coarse granularity prioritization techniques; (3) can the incorporation of measures of fault proneness into prioritization techniques improve their effectiveness? This paper reports the results of new experiments addressing these questions.

[1]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[2]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: A case study in a space application , 1999, J. Syst. Softw..

[3]  Gregg Rothermel,et al.  Aristotle: A System for Research on and Development of Program-Analysis-Based Tools , 1997 .

[4]  Lori A. Clarke,et al.  An information flow model of fault detection , 1993, ISSTA '93.

[5]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[6]  Thomas Ball,et al.  On the limit of control flow analysis for regression test selection , 1998, ISSTA '98.

[7]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[8]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[9]  David W. Binkley,et al.  Semantics Guided Regression Test Cost Reduction , 1997, IEEE Trans. Software Eng..

[10]  Sebastian G. Elbaum,et al.  Code churn: a measure for estimating the impact of code change , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[11]  Taghi M. Khoshgoftaar,et al.  Predicting Software Development Errors Using Software Complexity Metrics , 1990, IEEE J. Sel. Areas Commun..

[12]  Wei-Tek Tsai,et al.  Regression testing in an industrial environment , 1998, CACM.

[13]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[14]  Phyllis G. Frankl,et al.  Pythia: a regression test selection tool based on textual differencing , 1997 .

[15]  Tsong Yueh Chen,et al.  Dividing Strategies for the Optimization of a Test Suite , 1996, Inf. Process. Lett..

[16]  Thomas J. Ostrand,et al.  Automatic generation of test scripts from formal test specifications , 1989 .

[17]  Márcio Eduardo Delamaro,et al.  Proteum - A Tool for the Assessment of Test Adequacy for C Programs User's guide , 1996 .

[18]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[19]  Mary Lou Soffa,et al.  A methodology for controlling the size of a test suite , 1993, TSEM.

[20]  Albert L. Baker,et al.  A philosophy for software measurement , 1990, J. Syst. Softw..

[21]  John C. Munson.  Software measurement: Problems and practice , 1995, Ann. Softw. Eng..

[22]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[23]  R. Lyman Ott.,et al.  An introduction to statistical methods and data analysis , 1977 .

[24]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: a case study in a space application , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[25]  Tarak Goradia.  Dynamic impact analysis: a cost-effective technique to enforce error-propagation , 1993, ISSTA '93.

[26]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[27]  Thomas J. Ostrand,et al.  Automatic Generation of Test Scripts from Formal Test Specifications , 1989, Symposium on Testing, Analysis, and Verification.

[28]  Gregg Rothermel,et al.  An Experimental Determination of Su cient Mutation Operators , 1994 .

[29]  Phyllis G. Frankl,et al.  Empirical evaluation of the textual differencing regression testing technique , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[30]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[31]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[32]  John C. Munson,et al.  Software Evolution and the Fault Process , 1999 .

[33]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[34]  Lionel C. Briand,et al.  Investigating quality factors in object-oriented designs: an industrial case study , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[35]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[36]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[37]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[38]  David S. Rosenblum,et al.  TESTTUBE: a system for selective regression testing , 1994, Proceedings of 16th International Conference on Software Engineering.

[39]  Taghi M. Khoshgoftaar,et al.  The Detection of Fault-Prone Programs , 1992, IEEE Trans. Software Eng..

[40]  Richard G. Hamlet,et al.  Probable Correctness Theory , 1987, Inf. Process. Lett..

[41]  J. C. Munson,et al.  Software risk assessment through software measurement and modeling , 1998, 1998 IEEE Aerospace Conference Proceedings (Cat. No.98TH8339).

[42]  Jane T. Helwig.  SAS introductory guide , 1978 .

[43]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[44]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[45]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[46]  R. Freund,et al.  SAS for linear models : a guide to the ANOVA and GLM procedures , 1981 .

[47]  Hareton Leung,et al.  A study of integration testing and software regression at the integration level , 1990, Proceedings. Conference on Software Maintenance 1990.

[48]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[49]  Jeffrey M. Voas,et al.  PIE: A Dynamic Failure-Based Technique , 1992, IEEE Trans. Software Eng..

[50]  Elaine J. Weyuker,et al.  The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software , 1995, IEEE Trans. Software Eng..