Test suite reduction with selective redundancy

Software testing is a critical part of software development. Test suite sizes may grow significantly with subsequent modifications to the software over time. Due to time and resource constraints for testing, test suite minimization techniques attempt to remove those test cases from the test suite that have become redundant over time since the requirements covered by them are also covered by other test cases in the test suite. Prior work has shown that test suite minimization techniques can severely compromise the fault detection effectiveness of test suites. In this paper, we present a novel approach to test suite reduction that attempts to selectively keep redundant tests in the reduced suites. We implemented our technique by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the fault detection effectiveness of reduced suites without severely affecting the extent of test suite size reduction.

[1]  W. Eric Wong,et al.  Comparing the Fault Detection Effectiveness of Mutation and Data Flow Testing: An Empirical Study , 1993 .

[2]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[3]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[4]  Jonathan P. Bowen,et al.  Experimental evaluation of the variation in effectiveness for DC, FPC and MC/DC test criteria , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[5]  Hiralal Agrawal,et al.  Dominators, super blocks, and program coverage , 1994, POPL '94.

[6]  Michael D. Ernst,et al.  Improving test suites via operational abstraction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[7]  Nilesh Kawane.  Fault Detection Effectiveness of UML Design Model Test Adequacy Criteria , 2003 .

[8]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[9]  K. K. Aggarwal,et al.  Code coverage based technique for prioritizing test cases for regression testing , 2004, SOEN.

[10]  Vasek Chvátal,et al.  A Greedy Heuristic for the Set-Covering Problem , 1979, Math. Oper. Res..

[11]  Joseph Robert Horgan,et al.  Effect of test set size and block coverage on the fault detection effectiveness , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[12]  W. Eric Wong,et al.  Effect of Test Set Minimization on the Fault Detection Effectiveness of the All-Uses Criterion , 1994 .

[13]  Lori Pollock,et al.  A scalable approach to user-session based testing of Web applications through concept analysis , 2004 .

[14]  Larry J Morell,et al.  A Theory of Fault-Based Testing , 1990, IEEE Trans. Software Eng..

[15]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[16]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[17]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: a case study in a space application , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[18]  David Leon,et al.  A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[19]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[20]  Tatsuhiro Tsuchiya,et al.  On fault classes and error detection capability of specification-based testing , 2002, TSEM.

[21]  D. Richard Kuhn.  Fault classes and error detection capability of specification-based testing , 1999, TSEM.

[22]  Jane Huffman Hayes,et al.  Fault detection effectiveness of spathic test data , 2002, Eighth IEEE International Conference on Engineering of Complex Computer Systems, 2002. Proceedings..

[23]  Emily Hill,et al.  An empirical comparison of test suite reduction techniques for user-session-based testing of Web applications , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[24]  Joseph A. Morgan,et al.  Predicting fault detection effectiveness , 1997, Proceedings Fourth International Software Metrics Symposium.

[25]  Thomas J. Ostrand,et al.  Automatic Generation of Test Scripts from Formal Test Specifications , 1989, Symposium on Testing, Analysis, and Verification.

[26]  Hira Agrawal,et al.  Efficient coverage testing using global dominator graphs , 1999, ACM SIGSOFT Softw. Eng. Notes.

[27]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[28]  J. R. Horgan,et al.  A data flow coverage testing tool for C , 1992, [1992] Proceedings of the Second Symposium on Assessment of Quality Software Development Tools.

[29]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[30]  Tsong Yueh Chen,et al.  Dividing Strategies for the Optimization of a Test Suite , 1996, Inf. Process. Lett..

[31]  Phyllis G. Frankl,et al.  Further empirical studies of test effectiveness , 1998, SIGSOFT '98/FSE-6.

[32]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[33]  Phyllis G. Frankl,et al.  An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing , 1993, IEEE Trans. Software Eng..

[34]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[35]  T. Y. Chen,et al.  Heuristics Towards The Optimization Of TheSize Of A Test Suite , 1970 .

[36]  Jeffrey M. Voas,et al.  PIE: A Dynamic Failure-Based Technique , 1992, IEEE Trans. Software Eng..

[37]  Gregg Rothermel,et al.  Aristotle: A System for Research on and Development of Program-Analysis-Based Tools , 1997 .

[38]  Elaine J. Weyuker,et al.  A Formal Analysis of the Fault-Detecting Ability of Testing Methods , 1993, IEEE Trans. Software Eng..

[39]  Mats P. E. Heimdahl,et al.  Test-suite reduction for model based tests: effects on test quality and implications for testing , 2004 .

[40]  Gregg Rothermel,et al.  Experiments to Assess the Cost-Benefits of Test-Suite Reduction , 1999 .

[41]  Jeffery von Ronne,et al.  Test Suite Minimization an Empirical Investigation an Abstract of the Thesis of Test Suite Minimization an Empirical Investigation Acknowledgment , 2022 .

[42]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[43]  Hira Agrawal,et al.  Efficient coverage testing using global dominator graphs , 1999, PASTE '99.

[44]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[45]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[46]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[47]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..