Suitability analysis of CSP- and SMT-solvers for test case generation

Model-based testing allows automatic test case generation from models like Symbolic Input Output Transition Systems (SIOTS) describing the behavior of the System under Test (SUT). In order to create feasible test inputs a Constraint Satisfaction Problem (CSP)-solver or a Satisfiability Modulo Theory (SMT)-solver can be used. The test input creation complexity depends amongst others on the variable domains, the variable number and the test length. Since the capabilities of a test generation from a given SIOTS strongly depends on the used CSP- or SMT-solver, the most suitable one has to be selected. In this paper we benchmarked commonly used CSP- and SMT-solvers like Minion, Choco, Z3 or CVC4, regarding their time needed to solve constraints typical in model-based testing. The experimental results show that the compared solvers can differ vastly regarding the number of correctly solved and the time needed for a set of constraints. Since the solver performance depends on the characteristics of the constraint to be solved, no winner can be named. However, the provided results allow the selection of the most suitable solver for dedicated problem sets.

[1]  John Harrison,et al.  Handbook of Practical Logic and Automated Reasoning , 2009 .

[2]  Peter J. Stuckey,et al.  Philosophy of the MiniZinc challenge , 2010, Constraints.

[3]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[4]  Toby Walsh,et al.  CSPLIB: A Benchmark Library for Constraints , 1999, CP.

[5]  Ian P. Gent,et al.  Minion: A Fast Scalable Constraint Solver , 2006, ECAI.

[6]  Bernhard K. Aichernig,et al.  Compositional Random Testing Using Extended Symbolic Transition Systems , 2011, ICTSS.

[7]  Daniel Kroening,et al.  Decision Procedures - An Algorithmic Point of View , 2008, Texts in Theoretical Computer Science. An EATCS Series.

[8]  Patrice Godefroid,et al.  SAGE: Whitebox Fuzzing for Security Testing , 2012, ACM Queue.

[9]  Dejan Jovanović,et al.  Solving Non-linear Arithmetic , 2012, IJCAR.

[10]  Peter J. Stuckey,et al.  MiniZinc: Towards a Standard CP Modelling Language , 2007, CP.

[11]  Nicolas Beldiceanu,et al.  Constraint Logic Programming , 1997 .

[12]  Rina Dechter,et al.  Network-Based Heuristics for Constraint-Satisfaction Problems , 1987, Artif. Intell..

[13]  Lakhdar Sais,et al.  Boosting Systematic Search by Weighting Constraints , 2004, ECAI.

[14]  Lars Kotthoff.  Constraint solvers: An empirical evaluation of design decisions , 2010, ArXiv.

[15]  Bernhard K. Aichernig,et al.  Generating Distinguishing Tests Using the Minion Constraint Solver , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[16]  Aaron Stump,et al.  SMT-COMP: Satisfiability Modulo Theories Competition , 2005, CAV.

[17]  J. Hooker.  Constraint programming , 2003 .

[18]  Peter Zoeteweij,et al.  An Analysis of Arithmetic Constraints on Integer Intervals , 2007, Constraints.

[19]  Miguel A. Salido,et al.  A non-binary constraint ordering heuristic for constraint satisfaction problems , 2008, Appl. Math. Comput..

[20]  Cesare Tinelli,et al.  Solving SAT and SAT Modulo Theories: From an abstract Davis--Putnam--Logemann--Loveland procedure to DPLL(T) , 2006, JACM.

[21]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[22]  Leonardo Mendonça de Moura,et al.  Solving non-linear arithmetic , 2012, ACCA.

[23]  Clark W. Barrett,et al.  The SMT-LIB Standard Version 2.0 , 2010 .

[24]  Michèle Sebag,et al.  Online Heuristic Selection in Constraint Programming , 2009 .

[25]  Eugene C. Freuder.  A Sufficient Condition for Backtrack-Free Search , 1982, JACM.

[26]  Philippe Herrmann,et al.  An Alternative to SAT-Based Approaches for Bit-Vectors , 2010, TACAS.