Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests

Context: In software development and maintenance, a software system may frequently be updated to meet rapidly changing user requirements. New test cases will be designed to ensure the correctness of new or modified functions, thus gradually increasing the test suite's size. Test suite reduction techniques aim to decrease the cost of regression testing by removing the redundant test cases from the test suite and then obtaining a representative set of test cases that still yield a high level of code coverage. Objective: Most of the existing reduction algorithms focus on decreasing the test suite's size. Yet, the differences in execution costs among test cases are usually significant and it may take a lot of execution time to run a test suite consisting of a few long-running test cases. This paper presents and empirically evaluates cost-aware algorithms that can produce the representative sets with lower execution costs. Method: We first use a cost-aware test case metric, called Irreplaceability, and its enhanced version, called EIrreplaceability, to evaluate the possibility that each test case can be replaced by others during test suite reduction. Furthermore, we construct a cost-aware framework that incorporates the concept of test irreplaceability into some well-known test suite reduction algorithms. Results: The effectiveness of the cost-aware framework is evaluated via the subject programs and test suites collected from the Software-artifact Infrastructure Repository - frequently chosen benchmarks for experimentally evaluating test suite reduction methods. The empirical results reveal that the presented algorithms produce representative sets that normally incur a low cost to yield a high level of test coverage. Conclusion: The presented techniques indeed enhance the capability of the traditional reduction algorithms to reduce the execution cost of a test suite. Especially for the additional Greedy algorithm, the presented techniques decrease the costs of the representative sets by 8.10-46.57%.

[1]  Michaela Huhn,et al.  Minimizing Test Execution Time During Test Generation , 2006, SET.

[2]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[3]  D. J. Robson.  Regression testing , 1993 .

[4]  Ali Selamat,et al.  Information and Software Technology , 2014 .

[5]  Hong Mei,et al.  An experimental comparison of four test suite reduction techniques , 2006, ICSE.

[6]  Giuliano Antoniol,et al.  Evolution and Search Based Metrics to Improve Defects Prediction , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[7]  David W. Binkley,et al.  Semantics Guided Regression Test Cost Reduction , 1997, IEEE Trans. Software Eng..

[8]  René Just,et al.  MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[9]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[10]  Mark Harman,et al.  Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation , 2010, J. Syst. Softw..

[11]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[12]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[13]  Tsong Yueh Chen,et al.  How well does test case prioritization integrate with statistical fault localization? , 2012, Inf. Softw. Technol..

[14]  Gregory M. Kapfhammer,et al.  Reducing the Cost of Regression Testing by Identifying Irreplaceable Test Cases , 2012, 2012 Sixth International Conference on Genetic and Evolutionary Computing.

[15]  Mary Lou Soffa,et al.  Efficient time-aware prioritization with knapsack solvers , 2007, WEASELTech '07.

[16]  Phyllis G. Frankl,et al.  Empirical evaluation of the textual differencing regression testing technique , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[17]  Neelam Gupta,et al.  A concept analysis inspired greedy algorithm for test suite minimization , 2005, PASTE '05.

[18]  Neelam Gupta,et al.  Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction , 2007, IEEE Transactions on Software Engineering.

[19]  Richard M. Karp,et al.  Reducibility among combinatorial problems" in complexity of computer computations , 1972 .

[20]  Chin-Yu Huang,et al.  Analysis of test suite reduction with enhanced tie-breaking techniques , 2009, Inf. Softw. Technol..

[21]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[22]  Andreas Zeller,et al.  Efficient unit test case minimization , 2007, ASE '07.

[23]  Gregory M. Kapfhammer,et al.  An empirical study of incorporating cost into test suite reduction and prioritization , 2009, SAC '09.

[24]  Donald Kossmann,et al.  A framework for efficient regression tests on database applications , 2007, The VLDB Journal.

[25]  Mary Lou Soffa,et al.  Database-aware test coverage monitoring , 2008, ISEC '08.

[26]  Gregory M. Kapfhammer,et al.  Localizing SQL faults in database applications , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[27]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[28]  Hyuncheol Park,et al.  Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing , 2008, 2008 Second International Conference on Secure System Integration and Reliability Improvement.

[29]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[30]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[31]  Alberto Sillitti,et al.  A method for characterizing energy consumption in Android smartphones , 2013, 2013 2nd International Workshop on Green and Sustainable Software (GREENS).

[32]  S. N. Sivanandam,et al.  A SURVEY ON AUTOMATIC TEST CASE GENERATION , 2011 .

[33]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[34]  C. G. Chung,et al.  An optimal representative set selection method , 2000, Inf. Softw. Technol..

[35]  Atif M. Memon,et al.  Call-Stack Coverage for GUI Test Suite Reduction , 2008, IEEE Transactions on Software Engineering.

[36]  Kuan-Li Peng,et al.  A history-based cost-cognizant test case prioritization technique in regression testing , 2012, J. Syst. Softw..

[37]  Cheng-qing Ye,et al.  A genetic algorithm for test-suite reduction , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[38]  Baowen Xu,et al.  An empirical study on the effectiveness of time-aware test case prioritization techniques , 2011, SAC.

[39]  Tsong Yueh Chen,et al.  A simulation study on some heuristics for test suite reduction , 1998, Inf. Softw. Technol..

[40]  LinChu-Ti,et al.  Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests , 2014 .

[41]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[42]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[43]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[44]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[45]  Gregg Rothermel,et al.  Cost-cognizant Test Case Prioritization , 2006 .

[46]  Per Runeson,et al.  A systematic review on regression test selection techniques , 2010, Inf. Softw. Technol..

[47]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[48]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[49]  Mustafa Bozkurt.  Cost-aware pareto optimal test suite minimisation for service-centric systems , 2013, GECCO '13.

[50]  Lu Zhang,et al.  An experimental study of four typical test suite reduction techniques , 2008, Inf. Softw. Technol..

[51]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[52]  Vasek Chvátal,et al.  A Greedy Heuristic for the Set-Covering Problem , 1979, Math. Oper. Res..

[53]  Chyan-Goei Chung,et al.  An enhanced zero-one optimal path set selection method , 1997, J. Syst. Softw..

[54]  John J. Marciniak,et al.  Encyclopedia of Software Engineering , 1994, Encyclopedia of Software Engineering.

[55]  Tsong Yueh Chen,et al.  A new heuristic for test suite reduction , 1998, Inf. Softw. Technol..

[56]  Hasan Ural,et al.  Test Suite Reduction Based on Dependence Analysis , 2006, ISCIS.