Studying Test Case Failure Prediction for Test Case Prioritization

Background: Test case prioritization refers to the process of ranking test cases within a test suite for execution. The goal is ranking fault revealing test cases higher so that in case of limited budget one only executes the top ranked tests and still detects as many bugs as possible. Since the actual fault detection ability of test cases is unknown before execution, heuristics such as "code coverage" of the test cases are used for ranking test cases. Other test quality metrics such as "coverage of the changed parts of the code" and "number of fails in the past"' have also been studied in the literature. Aims: In this paper, we propose using a logistic regression model to predict the failing test cases in the current release based on a set of test quality metrics. Method: We have studied the effect of including our newly proposed quality metric ("similarity-based" metric) into this model for tests prioritization. Results: The results of our experiments on five open source systems show that none of the individual quality metrics of our study outperforms the others in all the projects. Conclusions: However, the ranks given by the regression model are more consistent in prioritizing fault revealing test cases in the current release.

[1]  Hadi Hemmati,et al.  A similarity-based approach for test case prioritization using historical failure data , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[2]  Aditya P. Mathur,et al.  Foundations of Software Testing , 2007 .

[3]  Dan Gusfield,et al.  Algorithms on Strings, Trees, and Sequences - Computer Science and Computational Biology , 1997 .

[4]  K. Goulden,et al.  Effect Sizes for Research: A Broad Practical Approach , 2006 .

[5]  Hyunsook Do,et al.  Improving the effectiveness of test suite through mining historical data , 2014, MSR 2014.

[6]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[7]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[8]  John B. Goodenough,et al.  Toward a theory of test data selection , 1975 .

[9]  Hyuncheol Park,et al.  Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing , 2008, 2008 Second International Conference on Secure System Integration and Reliability Improvement.

[10]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[11]  Hadi Hemmati,et al.  Test case analytics: Mining test case traces to improve risk-driven testing , 2015, 2015 IEEE 1st International Workshop on Software Analytics (SWAN).

[12]  Hyunsook Do,et al.  Striving for Failure: An Industrial Case Study about Test Failure Prediction , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[13]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[14]  Sunita Sarawagi,et al.  Sequence Data Mining , 2005 .

[15]  Kuan-Li Peng,et al.  A history-based cost-cognizant test case prioritization technique in regression testing , 2012, J. Syst. Softw..

[16]  R. Grissom,et al.  Effect sizes for research: A broad practical approach. , 2005 .

[17]  Laurie A. Williams,et al.  Using In-Process Testing Metrics to Estimate Post-Release Field Quality , 2007, The 18th IEEE International Symposium on Software Reliability (ISSRE '07).

[18]  A. Zeller,et al.  Predicting Defects for Eclipse , 2007, Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007).

[19]  Michele Lanza,et al.  An extensive comparison of bug prediction approaches , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[20]  Andreas Zeller,et al.  Predicting faults from cached history , 2008, ISEC '08.

[21]  Atif M. Memon,et al.  Studying the Characteristics of a "Good" GUI Test Suite , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[22]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[23]  Mark Harman,et al.  Faster Fault Finding at Google Using Multi Objective Regression Test Optimisation , 2011 .

[24]  Mauro Pezzè,et al.  Software testing and analysis - process, principles and techniques , 2007 .

[25]  Stephen McCamant,et al.  The Daikon system for dynamic detection of likely invariants , 2007, Sci. Comput. Program..

[26]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[27]  Richard Torkar,et al.  Software fault prediction metrics: A systematic literature review , 2013, Inf. Softw. Technol..

[28]  Andrea Arcuri.  Longer is Better: On the Role of Test Sequence Length in Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[29]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[30]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[31]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[32]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[33]  Audris Mockus,et al.  Software Dependencies, Work Dependencies, and Their Impact on Failures , 2009, IEEE Transactions on Software Engineering.

[34]  Audris Mockus,et al.  High-impact defects: a study of breakage and surprise defects , 2011, ESEC/FSE '11.

[35]  Cem Kaner,et al.  Foundations of Software Testing , 2013 .