Comparing and combining test-suite reduction and regression test selection

Regression testing is widely used to check that changes made to software do not break existing functionality, but regression test suites grow, and running them fully can become costly. Researchers have proposed test-suite reduction and regression test selection as two approaches to reduce this cost by not running some of the tests from the test suite. However, previous research has not empirically evaluated how the two approaches compare to each other, and how well a combination of these approaches performs. We present the first extensive study that compares test-suite reduction and regression test selection approaches individually, and also evaluates a combination of the two approaches. We also propose a new criterion to measure the quality of tests with respect to software changes. Our experiments on 4,793 commits from 17 open-source projects show that regression test selection runs on average fewer tests (by 40.15pp) than test-suite reduction. However, test-suite reduction can have a high loss in fault-detection capability with respect to the changes, whereas a (safe) regression test selection has no loss. The experiments also show that a combination of the two approaches runs even fewer tests (on average 5.34pp) than regression test selection, but these tests still have a loss in fault-detection capability with respect to the changes.

[1]  Darko Marinov,et al.  Balancing trade-offs in test-suite reduction , 2014, SIGSOFT FSE.

[2]  David S. Rosenblum,et al.  Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies , 1997, IEEE Trans. Software Eng..

[3]  Sarfraz Khurshid,et al.  FaultTracer: a change impact and regression fault analysis tool for evolving Java programs , 2012, SIGSOFT FSE.

[4]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[5]  Darko Marinov,et al.  Ekstazi: Lightweight Test Selection , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[6]  Rupak Majumdar,et al.  Regression Test Selection for Distributed Software Histories , 2014, CAV.

[7]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[8]  Gregg Rothermel,et al.  Techniques for improving regression testing in continuous integration development environments , 2014, SIGSOFT FSE.

[9]  Jonathan I. Maletic,et al.  What's a Typical Commit? A Characterization of Open Source Software Repositories , 2008, 2008 16th IEEE International Conference on Program Comprehension.

[10]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[11]  Daniela E. Damian,et al.  The promises and perils of mining GitHub , 2009, MSR 2014.

[12]  Gregg Rothermel,et al.  On-demand test suite reduction , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[13]  Danny Dig,et al.  How do centralized and distributed version control systems impact software changes? , 2014, ICSE.

[14]  Neelam Gupta,et al.  A concept analysis inspired greedy algorithm for test suite minimization , 2005, PASTE '05.

[15]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[16]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[17]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[18]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[19]  Brendan Murphy,et al.  The Art of Testing Less without Sacrificing Quality , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[20]  Darko Marinov,et al.  Practical regression test selection with dynamic file dependencies , 2015, ISSTA.

[21]  I. Comparison Faster Mutation Testing Inspired by Test Prioritization and Reduction , 2013 .

[22]  Tsong Yueh Chen,et al.  A new heuristic for test suite reduction , 1998, Inf. Softw. Technol..

[23]  Gregg Rothermel,et al.  The impact of software evolution on code coverage information , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[24]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[25]  Bogdan Korel,et al.  Model based regression test reduction using dependence analysis , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[26]  Sarfraz Khurshid,et al.  An Empirical Study of JUnit Test-Suite Reduction , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[27]  Mark Harman,et al.  Faster Fault Finding at Google Using Multi Objective Regression Test Optimisation , 2011 .

[28]  Daniel M. Germán,et al.  The promises and perils of mining git , 2009, 2009 6th IEEE International Working Conference on Mining Software Repositories.

[29]  Ralph E. Johnson,et al.  Automated Detection of Refactorings in Evolving Components , 2006, ECOOP.

[30]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[31]  Mario Piattini,et al.  Mutation Testing , 2014, IEEE Software.

[32]  Tsong Yueh Chen,et al.  A simulation study on some heuristics for test suite reduction , 1998, Inf. Softw. Technol..

[33]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: A case study in a space application , 1999, J. Syst. Softw..

[34]  Abhik Roychoudhury,et al.  CoREBench: studying complexity of regression errors , 2014, ISSTA 2014.