Feedback-directed unit test generation for C/C++ using concolic execution

In industry, software testing and coverage-based metrics are the predominant techniques to check correctness of software. This paper addresses automatic unit test generation for programs written in C/C++. The main idea is to improve the coverage obtained by feedback-directed random test generation methods, by utilizing concolic execution on the generated test drivers. Furthermore, for programs with numeric computations, we employ non-linear solvers in a lazy manner to generate new test inputs. These techniques significantly improve the coverage provided by a feedback-directed random unit testing framework, while retaining the benefits of full automation. We have implemented these techniques in a prototype platform, and describe promising experimental results on a number of C/C++ open source benchmarks.

[1]  Gordon Fraser,et al.  Testing Container Classes: Random or Systematic? , 2011, FASE.

[2]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[3]  Marcelo d'Amorim,et al.  CORAL: Solving Complex Constraints for Symbolic PathFinder , 2011, NASA Formal Methods.

[4]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[5]  Gogul Balakrishnan,et al.  Interprocedural Exception Analysis for C++ , 2011, ECOOP.

[6]  Alessandro Orso,et al.  LEAKPOINT: pinpointing the causes of memory leaks , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[7]  Patrice Godefroid,et al.  Proving memory safety of floating-point computations by combining static and dynamic program analysis , 2010, ISSTA '10.

[8]  Sriram Sankaranarayanan,et al.  Object Model Construction for Inheritance in C++ and Its Applications to Program Analysis , 2012, CC.

[9]  Sarfraz Khurshid,et al.  ParSym: Parallel symbolic execution , 2010, 2010 2nd International Conference on Software Technology and Engineering.

[10]  Michael D. Ernst,et al.  Combined static and dynamic automated test generation , 2011, ISSTA '11.

[11]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[12]  Gregg Rothermel,et al.  A Scalable Distributed Concolic Testing Approach: An Empirical Evaluation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[13]  Rupak Majumdar,et al.  Systematic testing for control applications , 2010, Eighth ACM/IEEE International Conference on Formal Methods and Models for Codesign (MEMOCODE 2010).

[14]  Frédéric Benhamou,et al.  Algorithm 852: RealPaver: an interval solver using constraint satisfaction techniques , 2006, TOMS.

[15]  Thomas R. Gross,et al.  Ballerina: Automatic generation and clustering of efficient random unit tests for multithreaded code , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[16]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[17]  Sriram Sankaranarayanan,et al.  Modeling and Analyzing the Interaction of C and C++ Strings , 2011, FoVeOOS.

[18]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[19]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[20]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[21]  William R. Harris,et al.  Program analysis via satisfiability modulo path programs , 2010, POPL '10.

[22]  Corina S. Pasareanu,et al.  Symbolic execution with mixed concrete-symbolic solving , 2011, ISSTA '11.

[23]  Marcelo d'Amorim,et al.  Symbolic Execution with Interval Solving and Meta-heuristic Search , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[24]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[25]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[26]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[27]  Nikolai Tillmann,et al.  Automating Software Testing Using Program Analysis , 2008, IEEE Software.

[28]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[29]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[30]  Tao Xie,et al.  Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[31]  Sriram Sankaranarayanan,et al.  DC2: A framework for scalable, scope-bounded software verification , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[32]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[33]  George Candea,et al.  Parallel symbolic execution for automated real-world software testing , 2011, EuroSys '11.

[34]  David A. McAllester,et al.  Solving Polynomial Systems Using a Branch and Prune Approach , 1997 .

[35]  Luc Jaulin,et al.  Applied Interval Analysis , 2001, Springer London.