A Hybrid Directed Test Suite Augmentation Technique

Test suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. In previous work, we studied two approaches to augmentation, one using a concolic test case generation algorithm and one using a genetic test case generation algorithm. We found that these two approaches behaved quite differently in terms of their costs and their abilities to generate effective test cases for evolving programs. In this paper, we present a hybrid test suite augmentation technique that combines these two test case generation algorithms. We report the results of an empirical study that shows that this hybrid technique can be effective, but with varying degrees of costs, and we analyze our results further to provide suggestions for reducing costs.

[1]  Alessandro Orso,et al.  MATRIX: Maintenance-Oriented Testing Requirements Identifier and Examiner , 2006, Testing: Academic & Industrial Conference - Practice And Research Techniques (TAIC PART'06).

[2]  Stefan Wappler,et al.  Using evolutionary algorithms for the unit testing of object-oriented software , 2005, GECCO '05.

[3]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[4]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[5]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[6]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[7]  José Javier Dolado,et al.  A tabu search algorithm for structural software testing , 2008, Comput. Oper. Res..

[8]  Tao Xie,et al.  Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[9]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[10]  Alessandro Orso,et al.  Scaling regression testing to large software systems , 2004, SIGSOFT '04/FSE-12.

[11]  Alan Hartman,et al.  Model Driven Testing – AGEDIS Architecture Interfaces and Tools , 2003 .

[12]  Gregg Rothermel,et al.  Selecting tests and identifying test coverage requirements for modified software , 1994, ISSTA '94.

[13]  Sarfraz Khurshid,et al.  Directed incremental symbolic execution , 2011, PLDI '11.

[14]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[15]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[16]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[17]  Roy P. Pargas,et al.  Test-Data Generation Using Genetic Algorithms , 1999, Softw. Test. Verification Reliab..

[18]  Alessandro Orso,et al.  Test-Suite Augmentation for Evolving Software , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[19]  Zhihong Xu,et al.  Directed test suite augmentation , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[20]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[21]  David W. Binkley,et al.  Semantics Guided Regression Test Cost Reduction , 1997, IEEE Trans. Software Eng..

[22]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[23]  Wolfram Schulte,et al.  eXpress : Guided Path Exploration for Regression Test Generation , 2010 .

[24]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[25]  Nikolai Tillmann,et al.  eXpress: guided path exploration for efficient regression test generation , 2011, ISSTA '11.

[26]  Matthew B. Dwyer,et al.  Differential symbolic execution , 2008, SIGSOFT '08/FSE-16.

[27]  Tsong Yueh Chen,et al.  Quasi-Random Testing , 2005, IEEE Transactions on Reliability.

[28]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[29]  Robert S. Arnold,et al.  Software Change Impact Analysis , 1996 .

[30]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[31]  Myra B. Cohen,et al.  Factors affecting the use of genetic algorithms in test suite augmentation , 2010, GECCO '10.

[32]  Myra B. Cohen,et al.  Directed test suite augmentation: techniques and tradeoffs , 2010, FSE '10.

[33]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[34]  Rajiv Gupta,et al.  Program Slicing‐Based Regression Testing Techniques , 1996, Softw. Test. Verification Reliab..

[35]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.