Combining Static Analysis and Constraint Solving for Automatic Test Case Generation

We present an approach in automatic test generation that combines features of static analysis and bounded symbolic computation that is capable of producing a test suite that can be used to declare a program under test safe within bounds. We first use the results produced by static analysis which will identify a list of potential errors in the program. We restrict our search to the locations where errors can exist and aim to find exactly one test case per real bug. We have built a prototype tool (called Batg) that implements our approach. We report the results of running it on a number of benchmarks from well known benchmarking suites. We compare Batgto KLEE (an automatic test generation framework) and CBMC(a bounded model checker). This comparison is based on the time taken by the tools, the number of bugs found and the number of generated test cases. We analyse the results of our experiment, demonstrating the benefits of our approach.

[1]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[2]  Erica Mealy,et al.  BegBunch: benchmarking for C bug detection tools , 2009, DEFECTS '09.

[3]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[4]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[5]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[6]  Cristina Cifuentes.  Parfait - A Scalable Bug Checker for C Code , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[7]  Sarah Smith Heckman,et al.  On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques , 2008, ESEM '08.

[8]  Armin Biere,et al.  Bounded model checking , 2003, Adv. Comput..

[9]  Richard Lippmann,et al.  Testing static analysis tools using exploitable buffer overflows from open source code , 2004, SIGSOFT '04/FSE-12.

[10]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[11]  Marsha Chechik,et al.  A buffer overflow benchmark for software model checkers , 2007, ASE.

[12]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[13]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[14]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[15]  Padmanabhan Krishnan,et al.  Comparing model checking and static program analysis: A case study in error detection approaches , 2010 .

[16]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.