An Empirical Study of JUnit Test-Suite Reduction

As test suites grow larger during software evolution, regression testing becomes expensive. To reduce the cost of regression testing, test-suite reduction aims to select a minimal subset of the original test suite that can still satisfy all the test requirements. While traditional test-suite reduction techniques were intensively studied on C programs with specially generated test suites, there are limited studies for test-suite reduction on programs with real-world test suites. In this paper, we investigate test-suite reduction techniques on Java programs with real-world JUnit test suites. We implemented four representative test-suite reduction techniques for JUnit test suites. We performed an empirical study on 19 versions of four real-world Java programs, ranging from 1.89 KLoC to 80.44 KLoC. Our study investigates both the benefits and the costs of test-suite reduction. The results show that the four traditional test-suite reduction techniques can effectively reduce these JUnit test suites without substantially reducing their fault-detection capability. Based on the results, we provide a guideline for achieving cost-effective JUnit test suite reduction.

[1]  Neelam Gupta,et al.  Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction , 2007, IEEE Transactions on Software Engineering.

[2]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[3]  Khaled El-Fakih,et al.  Simulated Annealing and Genetic Algorithms for Optimal Regression Testing , 1999, J. Softw. Maintenance Res. Pract..

[4]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: A case study in a space application , 1999, J. Syst. Softw..

[5]  Neelam Gupta,et al.  Test suite reduction with selective redundancy , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[6]  Alessandro Orso,et al.  Regression test selection for Java software , 2001, OOPSLA '01.

[7]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[8]  J. R. Horgan,et al.  A data flow coverage testing tool for C , 1992, [1992] Proceedings of the Second Symposium on Assessment of Quality Software Development Tools.

[9]  Tsong Yueh Chen,et al.  A new heuristic for test suite reduction , 1998, Inf. Softw. Technol..

[10]  Alfred V. Aho,et al.  The Design and Analysis of Computer Algorithms , 1974 .

[11]  D. J. Robson,et al.  Revalidation during the software maintenance phase , 1989, Proceedings. Conference on Software Maintenance - 1989.

[12]  T. Y. Chen,et al.  Heuristics Towards The Optimization Of TheSize Of A Test Suite , 1970 .

[13]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[14]  Lu Zhang,et al.  An experimental study of four typical test suite reduction techniques , 2008, Inf. Softw. Technol..

[15]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[16]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[17]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[18]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[19]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[20]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[21]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[22]  Tsong Yueh Chen,et al.  A simulation study on some heuristics for test suite reduction , 1998, Inf. Softw. Technol..

[23]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[24]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[25]  Gregg Rothermel,et al.  Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis , 2006, Empirical Software Engineering.

[26]  Ted K. Ralphs,et al.  The Symphony Callable Library for Mixed Integer Programming , 2005 .

[27]  Lu Zhang,et al.  Prioritizing JUnit test cases in absence of coverage information , 2009, 2009 IEEE International Conference on Software Maintenance.

[28]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[29]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[30]  Alessandro Orso,et al.  MINTS: A general framework and tool for supporting test-suite minimization , 2009, 2009 IEEE 31st International Conference on Software Engineering.