How Does Regression Test Prioritization Perform in Real-World Software Evolution?

In recent years, researchers have intensively investigated various topics in test prioritization, which aims to re-order tests to increase the rate of fault detection during regression testing. While the main research focus in test prioritization is on proposing novel prioritization techniques and evaluating on more and larger subject systems, little effort has been put on investigating the threats to validity in existing work on test prioritization. One main threat to validity is that existing work mainly evaluates prioritization techniques based on simple artificial changes on the source code and tests. For example, the changes in the source code usually include only seeded program faults, whereas the test suite is usually not augmented at all. On the contrary, in real-world software development, software systems usually undergo various changes on the source code and test suite augmentation. Therefore, it is not clear whether the conclusions drawn by existing work in test prioritization from the artificial changes are still valid for real-world software evolution. In this paper, we present the first empirical study to investigate this important threat to validity in test prioritization. We reimplemented 24 variant techniques of both the traditional and time-aware test prioritization, and investigated the impacts of software evolution on those techniques based on the version history of 8 real-world Java programs from GitHub. The results show that for both traditional and time-aware test prioritization, test suite augmentation significantly hampers their effectiveness, whereas source code changes alone do not influence their effectiveness much.

[1]  Tao Xie,et al.  To Be Optimal or Not in Test-Case Prioritization , 2016, IEEE Transactions on Software Engineering.

[2]  Gregg Rothermel,et al.  An empirical study of the effect of time constraints on the cost-benefits of regression testing , 2008, SIGSOFT '08/FSE-16.

[3]  Darko Marinov,et al.  Comparing and combining test-suite reduction and regression test selection , 2015, ESEC/SIGSOFT FSE.

[4]  Gregg Rothermel,et al.  A controlled experiment assessing test case prioritization techniques via mutation faults , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[5]  Darko Marinov,et al.  Balancing trade-offs in test-suite reduction , 2014, SIGSOFT FSE.

[6]  Dianxiang Xu,et al.  Prioritizing State-Based Aspect Tests , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[7]  Gregg Rothermel,et al.  An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models , 2006, SIGSOFT '06/FSE-14.

[8]  Cem Kaner.  Improving the maintainability of automated test suites , 1997 .

[9]  Lu Zhang,et al.  Mutation-based test-case prioritization in software evolution , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[10]  Paolo Tonella,et al.  Using the Case-Based Ranking Methodology for Test Case Prioritization , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[11]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[12]  Alex Groce,et al.  Using test case reduction and prioritization to improve symbolic execution , 2014, ISSTA 2014.

[13]  Sarfraz Khurshid,et al.  An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[14]  Zhiqiang Lin,et al.  Half a Century of Practice: Who Is Still Storing Plaintext Passwords? , 2015, ISPEC.

[15]  Sarfraz Khurshid,et al.  FaultTracer: a change impact and regression fault analysis tool for evolving Java programs , 2012, SIGSOFT FSE.

[16]  Mary Jean Harrold,et al.  Testing evolving software , 1999, J. Syst. Softw..

[17]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[18]  Lu Zhang,et al.  Test Case Prioritization for Compilers: A Text-Vector Based Approach , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[19]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[20]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[21]  Gregg Rothermel,et al.  The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments , 2010, IEEE Transactions on Software Engineering.

[22]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[23]  Lu Zhang,et al.  Prioritizing JUnit test cases in absence of coverage information , 2009, 2009 IEEE International Conference on Software Maintenance.

[24]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[25]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[26]  Paolo Tonella,et al.  Test Case Prioritization for Audit Testing of Evolving Web Services Using Information Retrieval Techniques , 2011, 2011 IEEE International Conference on Web Services.

[27]  Thomas H. Wonnacott,et al.  Introductory Statistics , 2007, Technometrics.

[28]  Sarfraz Khurshid,et al.  Localizing failure-inducing program edits based on spectrum information , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[29]  Ahmed E. Hassan,et al.  Static test case prioritization using topic models , 2014, Empirical Software Engineering.

[30]  Mary Jean Harrold,et al.  Recomputing Coverage Information to Assist Regression Testing , 2009, IEEE Transactions on Software Engineering.

[31]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[32]  Hyuncheol Park,et al.  Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing , 2008, 2008 Second International Conference on Secure System Integration and Reliability Improvement.

[33]  References , 1971 .

[34]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[35]  Tim Miller,et al.  Using Dependency Structures for Prioritization of Functional Test Suites , 2013, IEEE Transactions on Software Engineering.

[36]  Tao Xie,et al.  Is operator-based mutant selection superior to random mutant selection? , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[37]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[38]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[39]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[40]  Darko Marinov,et al.  Reliable testing: detecting state-polluting tests to prevent test dependency , 2015, ISSTA.

[41]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[42]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[43]  Gregg Rothermel,et al.  A Static Approach to Prioritizing JUnit Test Cases , 2012, IEEE Transactions on Software Engineering.

[44]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[45]  Thomas Ball,et al.  On the limit of control flow analysis for regression test selection , 1998, ISSTA '98.

[46]  Gregg Rothermel,et al.  Bridging the gap between the total and additional test-case prioritization strategies , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[47]  Gregg Rothermel,et al.  A Unified Test Case Prioritization Approach , 2014, ACM Trans. Softw. Eng. Methodol..

[48]  Mary Lou Soffa,et al.  Efficient time-aware prioritization with knapsack solvers , 2007, WEASELTech '07.

[49]  Gregg Rothermel,et al.  Empirical studies of test case prioritization in a JUnit testing environment , 2004, 15th International Symposium on Software Reliability Engineering.

[50]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[51]  Yuhua Qi,et al.  The strength of random search on automated program repair , 2014, ICSE.

[52]  Gregg Rothermel,et al.  Using sensitivity analysis to create simplified economic models for regression testing , 2008, ISSTA '08.

[53]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[54]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[55]  T. H. Tse,et al.  Data flow testing of service-oriented workflow applications , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[56]  Sarfraz Khurshid,et al.  An Empirical Study of JUnit Test-Suite Reduction , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[57]  RothermelGregg,et al.  A safe, efficient regression test selection technique , 1997 .

[58]  Tao Xie,et al.  Quota-constrained test-case prioritization for regression testing of service-centric systems , 2008, 2008 IEEE International Conference on Software Maintenance.

[59]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[60]  I. Comparison.  Faster Mutation Testing Inspired by Test Prioritization and Reduction , 2013 .

[61]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[62]  Günther Ruhe,et al.  Search Based Software Engineering , 2013, Lecture Notes in Computer Science.

[63]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[64]  Mark Harman,et al.  Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge , 2009, ISSTA.

[65]  Gregg Rothermel,et al.  Cost-cognizant Test Case Prioritization , 2006 .

[66]  T. H. Tse,et al.  Test case prioritization for regression testing of service-oriented business applications , 2009, WWW '09.