Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity

Test suite reduction is an important approach that decreases the cost of regression testing. A test suite reduction technique operates based on the relationship between the test cases in the regression test suite and the test requirements in the program under test. Thus, its effectiveness should be closely related to the complexity of a regression test suite the product of the number of test cases and the number of test requirements. Our previous work has shown that cost-aware techniques (i.e., the test suite reduction techniques that aim to decrease the regression test suite's execution cost) generally outperform the others in terms of decreasing the cost of running the regression test suite. However, the previous empirical studies that evaluated cost-aware techniques did not take into account test suite complexity. That is, prior experiments do not reveal if the cost-aware techniques scale and work effectively on test suites with more test cases and more test requirements. This means that prior experiments do not appropriately shed light on how well test suite reduction methods work with large programs or test suites. Therefore, this paper focuses on the Greedy-based techniques and empirically evaluates the additional Greedy and two cost-aware Greedy techniques at different levels of test suite complexity from various standpoints including the cost taken to run the regression test suite, the time taken to reduce the test suites, the total regression testing costs, the fault detection capability, the fault detection efficiency, and the common rates of the representative sets. To the best of our knowledge, none of the previous empirical studies classify a considerable number of test suites according to their complexity. Nor do any prior experiments evaluate the test suite reduction techniques, in terms of the aforementioned criteria, at different levels of test suite complexity. This paper represents the first such attempt to carry out this important task. Based on the empirical results, we confirm the strengths and weaknesses of the cost-aware techniques and develop insights into how the cost-aware techniques' effectiveness varies as the test suite complexity increases. We evaluate test reduction methods at various levels of test suite complexity.Few previous studies evaluated the test reduction methods in the way that we did.The cost-aware methods generally attain the lowest total regression testing costs.The cost-aware methods generally realize higher fault detection efficiency.The benefits of using cost-aware methods increase as test suite complexity grows.

[1]  Cheng-qing Ye,et al.  A genetic algorithm for test-suite reduction , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[2]  Gregory M. Kapfhammer,et al.  An empirical study of incorporating cost into test suite reduction and prioritization , 2009, SAC '09.

[3]  Kuan-Li Peng,et al.  A history-based cost-cognizant test case prioritization technique in regression testing , 2012, J. Syst. Softw..

[4]  Gregg Rothermel,et al.  A Static Approach to Prioritizing JUnit Test Cases , 2012, IEEE Transactions on Software Engineering.

[5]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[6]  Arnaud Gotlieb,et al.  Cost-effective test suite minimization in product lines using search techniques , 2015, J. Syst. Softw..

[7]  Neelam Gupta,et al.  Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction , 2007, IEEE Transactions on Software Engineering.

[8]  Saeed Parsa,et al.  Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles , 2009, CEE-SET.

[9]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[10]  Haiyang Sun,et al.  Comprehensive Multiplatform Dynamic Program Analysis for Java and Android , 2016, IEEE Software.

[11]  Leonardo Mariani,et al.  MASH: tool integration made easy , 2013, Softw. Pract. Exp..

[12]  RothermelGregg,et al.  Prioritizing JUnit Test Cases , 2006 .

[13]  Alex Groce,et al.  Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites , 2015, ACM Trans. Softw. Eng. Methodol..

[14]  Atul Gupta,et al.  An Experimental Comparison of the Effectiveness of Control Flow Based Testing Approaches on Seeded Faults , 2006, TACAS.

[15]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: a case study in a space application , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[16]  Phyllis G. Frankl,et al.  Empirical evaluation of the textual differencing regression testing technique , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[17]  Neelam Gupta,et al.  A concept analysis inspired greedy algorithm for test suite minimization , 2005, PASTE '05.

[18]  Alessandro Orso,et al.  MINTS: A general framework and tool for supporting test-suite minimization , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[19]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[20]  Gregory M. Kapfhammer Regression Testing , 2010, Encyclopedia of Software Engineering.

[21]  Antonia Bertolino,et al.  Using Spanning Sets for Coverage Testing , 2003, IEEE Trans. Software Eng..

[22]  James A. Whittaker,et al.  Software Engineering is Not Enough , 2002, IEEE Softw..

[23]  Neelam Gupta,et al.  Test suite reduction with selective redundancy , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[24]  Behrouz Homayoun Far,et al.  Scalability improvement in software evaluation methodologies , 2009, 2009 IEEE International Conference on Information Reuse & Integration.

[25]  Mark Harman,et al.  Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation , 2010, J. Syst. Softw..

[26]  Roger S. Pressman,et al.  Software engineering (3rd ed.): a practitioner's approach , 1992 .

[27]  T. H. Tse,et al.  Fault localization through evaluation sequences , 2010, J. Syst. Softw..

[28]  Mustafa Bozkurt Cost-aware pareto optimal test suite minimisation for service-centric systems , 2013, GECCO '13.

[29]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[30]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[31]  Lu Zhang,et al.  An experimental study of four typical test suite reduction techniques , 2008, Inf. Softw. Technol..

[32]  Atif M. Memon,et al.  Call stack coverage for test suite reduction , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[33]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[34]  Rayford B. Vaughn,et al.  Software requirement understanding using Pathfinder networks: discovering and evaluating mental models , 2005, J. Syst. Softw..

[35]  Mark Harman,et al.  Highly Scalable Multi Objective Test Suite Minimisation Using Graphics Cards , 2011, SSBSE.

[36]  Chin-Yu Huang,et al.  Analysis of test suite reduction with enhanced tie-breaking techniques , 2009, Inf. Softw. Technol..

[37]  Alex Groce,et al.  Code coverage for suite evaluation by developers , 2014, ICSE.

[38]  Mohammad Abdollahi Azgomi,et al.  An improved method for test case prioritization by incorporating historical test case data , 2012, Sci. Comput. Program..

[39]  Atif M. Memon,et al.  Call-Stack Coverage for GUI Test Suite Reduction , 2008, IEEE Trans. Software Eng..

[40]  Vasek Chvátal,et al.  A Greedy Heuristic for the Set-Covering Problem , 1979, Math. Oper. Res..

[41]  Rainer Koschke,et al.  Journal of Software Maintenance and Evolution: Research and Practice Software Visualization in Software Maintenance, Reverse Engineering, and Re-engineering: a Research Survey , 2022 .

[42]  Bruce McMillin,et al.  Software engineering: What is it? , 2018, 2018 IEEE Aerospace Conference.

[43]  Gregory M. Kapfhammer,et al.  Reducing the Cost of Regression Testing by Identifying Irreplaceable Test Cases , 2012, 2012 Sixth International Conference on Genetic and Evolutionary Computing.

[44]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[45]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[46]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[47]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[48]  Gregg Rothermel,et al.  Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis , 2006, Empirical Software Engineering.

[49]  Ondrej Lhoták,et al.  Model checking of concurrent programs with static analysis of field accesses , 2015, Sci. Comput. Program..

[50]  Mary Lou Soffa,et al.  Test suite reduction and prioritization with call trees , 2007, ASE.

[51]  David W. Binkley,et al.  Semantics Guided Regression Test Cost Reduction , 1997, IEEE Trans. Software Eng..

[52]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[53]  Michael W. Godfrey A Hybrid Program Model for Object-Oriented Reverse Engineering , 2007, 15th IEEE International Conference on Program Comprehension (ICPC '07).

[54]  Gregory M. Kapfhammer,et al.  Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests , 2014, Inf. Softw. Technol..

[55]  Per Runeson,et al.  A systematic review on regression test selection techniques , 2010, Inf. Softw. Technol..

[56]  Ladan Tahvildari,et al.  Size-Constrained Regression Test Case Selection Using Multicriteria Optimization , 2012, IEEE Transactions on Software Engineering.

[57]  Alexandre Bergel,et al.  Increasing test coverage with Hapao , 2014, Sci. Comput. Program..

[58]  Lu Zhang,et al.  An Empirical Study on the Scalability of Selective Mutation Testing , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[59]  Mary Jean Harrold,et al.  Test-suite reduction and prioritization for modified condition/decision coverage , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[60]  Emily Hill,et al.  An empirical comparison of test suite reduction techniques for user-session-based testing of Web applications , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[61]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[62]  Wang Tiantian,et al.  A test-suite reduction approach to improving fault-localization effectiveness , 2013 .

[63]  Hong Mei,et al.  An experimental comparison of four test suite reduction techniques , 2006, ICSE.

[64]  Tsong Yueh Chen,et al.  A new heuristic for test suite reduction , 1998, Inf. Softw. Technol..

[65]  Andreas Podelski,et al.  Reducing GUI test suites via program slicing , 2014, ISSTA 2014.

[66]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .