Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet?

Whilst there is much evidence that both concolic and search based testing can outperform random testing, there has been little work demonstrating the effectiveness of either technique with complete real world software applications. As a consequence, many researchers have doubts not only about the scalability of both approaches but also their applicability to production code. This paper performs an empirical study applying a concolic tool, CUTE, and a search based tool, AUSTIN, to the source code of four large open source applications. Each tool is applied `out of the box'; that is without writing additional code for special handling of any of the individual subjects, or by tuning the tools' parameters. Perhaps surprisingly, the results show that both tools can only obtain at best a modest level of code coverage. Several challenges remain for improving automated test data generators in order to achieve higher levels of code coverage.

[1]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[2]  Patrice Godefroid.  Random testing for security: blackbox vs. whitebox fuzzing , 2007, RT '07.

[3]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[4]  Mark Harman,et al.  Meta-heuristic Search Algorithms for Regression Test Case Prioritization , 2007 .

[5]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[6]  McMinnPhil.  Search-based software test data generation: a survey , 2004 .

[7]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[8]  Mark Harman,et al.  Handling dynamic data structures in search based testing , 2008, GECCO '08.

[9]  Joachim Wegener,et al.  A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints , 2004, Real-Time Systems.

[10]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999, Softw. Test. Verification Reliab..

[11]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[12]  Dawson R. Engler,et al.  Execution Generated Test Cases: How to Make Systems Code Crash Itself , 2005, SPIN.

[13]  Tao Xie,et al.  Evacon: a framework for integrating evolutionary and concolic testing for object-oriented programs , 2007, ASE.

[14]  Y. L. Traon,et al.  From genetic to bacteriological algorithms for mutation‐based testing , 2005, Softw. Test. Verification Reliab..

[15]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[16]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[17]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[18]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[19]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[20]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[21]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[22]  Patrice Godefroid,et al.  Software partitioning for effective automated unit testing , 2006, EMSOFT '06.

[23]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[24]  Bruno Marre,et al.  PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis , 2005, EDCC.

[25]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[26]  Mark Harman,et al.  A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation , 2007, ISSTA '07.

[27]  Phil McMinn,et al.  IGUANA: Input Generation Using Automated Novel Algorithms. A Plug and Play Research Tool , 2007 .

[28]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[29]  John A. Clark,et al.  Formulating software engineering as a search problem , 2003, IEE Proc. Softw..

[30]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[31]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[32]  Nikolai Tillmann,et al.  DySy: dynamic symbolic execution for invariant inference , 2008, ICSE.

[33]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[34]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[35]  Mark Harman,et al.  The species per path approach to SearchBased test data generation , 2006, ISSTA '06.

[36]  Oliver Buehler,et al.  Evolutionary Functional Testing of an Automated Parking System , 2003 .

[37]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.