Call stack coverage for test suite reduction

Test suite reduction is an important test maintenance activity that attempts to reduce the size of a test suite with respect to some criteria. Emerging trends in software development such as component reuse, multi-language implementations, and stringent performance requirements present new challenges for existing reduction techniques that may limit their applicability. A test suite reduction technique that is not affected by these challenges is presented; it is based on dynamically generated language-independent information that can be collected with little run-time overhead. Specifically, test cases from the suite being reduced are executed on the application under test and the call stacks produced during execution are recorded. These call stacks are then used as a coverage requirement in a test suite reduction algorithm. Results of experiments on test suites for the space antenna-steering application show significant reduction in test suite size at the cost of a moderate loss in fault detection effectiveness.

[1]  Thomas Ball,et al.  The concept of dynamic analysis , 1999, ESEC/FSE-7.

[2]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: A case study in a space application , 1999, J. Syst. Softw..

[3]  Maurizio Morisio,et al.  Investigating and improving a COTS-based software development process , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[4]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[5]  Neelam Gupta,et al.  Test suite reduction with selective redundancy , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[6]  Atif M. Memon,et al.  Automating regression testing for evolving GUI software , 2005, J. Softw. Maintenance Res. Pract..

[7]  Douglas C. Schmidt,et al.  Skoll: distributed continuous quality assurance , 2004, Proceedings. 26th International Conference on Software Engineering.

[8]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[9]  Mary Lou Soffa,et al.  Coverage criteria for GUI testing , 2001, ESEC/FSE-9.

[10]  James R. Larus,et al.  Exploiting hardware performance counters with flow and context sensitive profiling , 1997, PLDI '97.

[11]  Michael D. Ernst,et al.  Improving test suites via operational abstraction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[12]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[13]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[14]  David Thomas,et al.  Programming Ruby - the pragmatic programmer's guide (2. ed.) , 2005 .

[15]  Galen C. Hunt,et al.  Detours: binary interception of Win32 functions , 1999 .

[16]  Phyllis G. Frankl,et al.  Further empirical studies of test effectiveness , 1998, SIGSOFT '98/FSE-6.

[17]  Gregg Rothermel,et al.  An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models , 2006, SIGSOFT '06/FSE-14.

[18]  Mary Lou Soffa,et al.  Hierarchical GUI Test Case Generation Using Automated Planning , 2001, IEEE Trans. Software Eng..

[19]  Stefan Wagner.  A model and sensitivity analysis of the quality economics of defect-detection techniques , 2006, ISSTA '06.

[20]  Lori Pollock,et al.  A scalable approach to user-session based testing of Web applications through concept analysis , 2004 .

[21]  David Notkin,et al.  Rostra: a framework for detecting redundant object-oriented unit tests , 2004 .

[22]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[23]  S. N. Weiss,et al.  All-Uses versus Mutation Testing : An ExperimentalComparison of E ectiveness , 1996 .

[24]  David Leon,et al.  Finding failures by cluster analysis of execution profiles , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[25]  Joseph Robert Horgan,et al.  Data flow coverage and the C language , 1991, TAV4.

[26]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[27]  David Thomas,et al.  Programming Ruby: the pragmatic programmer's guide , 2000 .

[28]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[29]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[30]  Atif M. Memon,et al.  Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software , 2005, IEEE Transactions on Software Engineering.

[31]  David Leon,et al.  A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[32]  Chengyun Chu,et al.  Test case prioritization , 1999 .

[33]  Mary Lou Soffa,et al.  Automated test oracles for GUIs , 2000, SIGSOFT '00/FSE-8.

[34]  Atanas Rountev,et al.  Static and dynamic analysis of call chains in java , 2004, ISSTA '04.

[35]  Joseph Robert Horgan,et al.  Effect of test set size and block coverage on the fault detection effectiveness , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[36]  Emily Hill,et al.  Web Application Testing with Customized Test Requirements - An Experimental Comparison Study , 2006, 2006 17th International Symposium on Software Reliability Engineering.