Multi-solver Support in Symbolic Execution

One of the main challenges of dynamic symbolic execution--an automated program analysis technique which has been successfully employed to test a variety of software--is constraint solving. A key decision in the design of a symbolic execution tool is the choice of a constraint solver. While different solvers have different strengths, for most queries, it is not possible to tell in advance which solver will perform better. In this paper, we argue that symbolic execution tools can, and should, make use of multiple constraint solvers. These solvers can be run competitively in parallel, with the symbolic execution engine using the result from the best-performing solver. We present empirical data obtained by running the symbolic execution engine KLEE on a set of real programs, and use it to highlight several important characteristics of the constraint solving queries generated during symbolic execution. In particular, we show the importance of constraint caching and counterexample values on the (relative) performance of KLEE configured to use different SMT solvers. We have implemented multi-solver support in KLEE, using the metaSMT framework, and explored how different state-of-the-art solvers compare on a large set of constraint-solving queries. We also report on our ongoing experience building a parallel portfolio solver in KLEE.

[1]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[2]  Stefan Frehse,et al.  metaSMT: Focus on Your Application not on Solver Integration , 2011, DIFTS@FMCAD.

[3]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[4]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[5]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[6]  Thomas R. Gross,et al.  Variant-based competitive parallel execution of sequential programs , 2010 .

[7]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[8]  Lakhdar Sais,et al.  ManySAT: a Parallel SAT Solver , 2009, J. Satisf. Boolean Model. Comput..

[9]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[10]  Cristian Cadar,et al.  make test-zesti: A symbolic execution solution for improving regression testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[11]  Matthew B. Dwyer,et al.  Green: reducing, reusing and recycling constraints in program analysis , 2012, SIGSOFT FSE.

[12]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[13]  Per Stenström,et al.  Semantic information based speculative parallel execution , 2010 .

[14]  Sarfraz Khurshid,et al.  Memoized symbolic execution , 2012, ISSTA 2012.

[15]  David L. Dill,et al.  A Decision Procedure for Bit-Vectors and Arrays , 2007, CAV.

[16]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[17]  Santosh Pande,et al.  Opportunistic computing: a new paradigm for scalable realism on many-cores , 2009 .

[18]  Youssef Hamadi,et al.  A Concurrent Portfolio Approach to SMT Solving , 2009, CAV.

[19]  Bernd Fischer,et al.  SMT-Based Bounded Model Checking for Embedded ANSI-C Software , 2012, IEEE Transactions on Software Engineering.

[20]  Alexander L. Wolf,et al.  Multiplicity computing: a vision of software engineering for next-generation computing platform applications , 2010, FoSER '10.

[21]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[22]  Nikolaj Bjørner,et al.  Satisfiability modulo theories , 2011, Commun. ACM.

[23]  Armin Biere,et al.  Boolector: An Efficient SMT Solver for Bit-Vectors and Arrays , 2009, TACAS.

[24]  C. Tinelli,et al.  The SMT-LIB Format: An Initial Proposal , 2003 .

[25]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).