Automated Testing and Debugging of SAT and QBF Solvers

Robustness and correctness are essential criteria for SAT and QBF solvers. We develop automated testing and debugging techniques designed and optimized for SAT and QBF solver development. Our fuzz testing techniques are able to find critical solver defects that lead to crashes, invalid satisfying assignments and incorrect satisfiability results. Moreover, we show that sequential and concurrent delta debugging techniques are highly effective in minimizing failure-inducing inputs.

[1]  Cesare Tinelli,et al.  Handbook of Satisfiability , 2021, Handbook of Satisfiability.

[2]  Armin Biere,et al.  Fuzzing and delta-debugging SMT solvers , 2009, SMT '09.

[3]  Armin Biere,et al.  PicoSAT Essentials , 2008, J. Satisf. Boolean Model. Comput..

[4]  Jared D. DeMott,et al.  Fuzzing for Software Security Testing and Quality Assurance , 2008 .

[5]  Martin Gebser,et al.  The Conflict-Driven Answer Set Solver clasp: Progress Report , 2009, LPNMR.

[6]  Toby Walsh,et al.  Beyond NP: the QSAT phase transition , 1999, AAAI/IAAI.

[7]  Sharad Malik,et al.  Validating SAT solvers using an independent resolution-based checker: practical implementations and other applications , 2003, 2003 Design, Automation and Test in Europe Conference and Exhibition.

[8]  Zhendong Su,et al.  HDD: hierarchical delta debugging , 2006, ICSE.

[9]  Peter van Beek,et al.  Principles and Practice of Constraint Programming - CP 2005, 11th International Conference, CP 2005, Sitges, Spain, October 1-5, 2005, Proceedings , 2005, CP.

[10]  Armin Biere,et al.  Resolve and Expand , 2004, SAT.

[11]  David M. Pennock,et al.  Exploiting a Theory of Phase Transitions in Three-Satisfiability Problems , 1996, AAAI/IAAI, Vol. 1.

[12]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[13]  Nadia Creignou,et al.  New Results on the Phase Transition for Random Quantified Boolean Formulas , 2008, SAT.

[14]  Wei Li,et al.  The SAT phase transition , 1999, ArXiv.

[15]  Luca Pulina,et al.  Evaluating and certifying QBFs: A comparison of state-of-the-art tools , 2009, AI Commun..

[16]  Allen Van Gelder.  Extracting (Easily) Checkable Proofs from a Satisfiability Solver that Employs both Preorder and Postorder Resolution , 2002, ISAIM.

[17]  Pedram Amini,et al.  Fuzzing: Brute Force Vulnerability Discovery , 2007 .

[18]  Barton P. Miller,et al.  Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services , 1995 .

[19]  Andreas Zeller,et al.  Why Programs Fail: A Guide to Systematic Debugging , 2005 .

[20]  Daniel Kroening,et al.  A First Step Towards a Unified Proof Checker for QBF , 2007, SAT.

[21]  S. Malik,et al.  Towards symmetric treatment of con?icts and satisfaction in quanti-fied Boolean satisfiability solv , 2002 .

[22]  Marco Schaerf,et al.  An Algorithm to Evaluate Quantified Boolean Formulae and Its Experimental Evaluation , 2002, Journal of Automated Reasoning.

[23]  J. Freeman.  Improvements to propositional satisfiability search algorithms , 1995 .

[24]  Marijn Heule.  SmArT solving: tools and techniques for satisfiability solvers , 2008 .

[25]  Yannet Interian,et al.  A Model for Generating Random Quantified Boolean Formulas , 2005, IJCAI.

[26]  Lakhdar Sais,et al.  ManySAT: a Parallel SAT Solver , 2009, J. Satisf. Boolean Model. Comput..

[27]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[28]  Fahiem Bacchus,et al.  Using SAT in QBF , 2005, CP.

[29]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[30]  Sharad Malik,et al.  Towards a Symmetric Treatment of Satisfaction and Conflicts in Quantified Boolean Formula Evaluation , 2002, CP.

[31]  Oliver Kullmann,et al.  Theory and Applications of Satisfiability Testing - SAT 2009, 12th International Conference, SAT 2009, Swansea, UK, June 30 - July 3, 2009. Proceedings , 2009, SAT.

[32]  Ke Xu,et al.  A Simple Model to Generate Hard Satisfiable Instances , 2005, IJCAI.

[33]  Hans Kleine Büning,et al.  Theory and Applications of Satisfiability Testing - SAT 2008, 11th International Conference, SAT 2008, Guangzhou, China, May 12-15, 2008. Proceedings , 2008, SAT.

[34]  Moshe Y. Vardi,et al.  Symbolic Decision Procedures for QBF , 2004, CP.

[35]  Bernd Becker,et al.  Multithreaded SAT Solving , 2007, 2007 Asia and South Pacific Design Automation Conference.

[36]  Adnan Darwiche,et al.  RSat 2.0: SAT Solver Description , 2006 .

[37]  Babić Domagoj,et al.  Exploiting structure for scalable software verification , 2008 .

[38]  Reinhold Letz,et al.  Lemma and Model Caching in Decision Procedures for Quantified Boolean Formulas , 2002, TABLEAUX.

[39]  Marco Benedetti,et al.  sKizzo: A Suite to Evaluate and Certify QBFs , 2005, CADE.

[40]  Andreas Zeller.  Why Programs Fail , 2005 .

[41]  Donald W. Loveland,et al.  A machine program for theorem-proving , 2011, CACM.

[42]  Cesare Tinelli,et al.  The SMT-LIB Standard: Version 1.2 , 2005 .

[43]  Sheila A. McIlraith,et al.  VARSAT: Integrating Novel Probabilistic Inference Techniques with DPLL Search , 2009, SAT.

[44]  Carlos Ansótegui,et al.  Generating Hard SAT/CSP Instances Using Expander Graphs , 2008, AAAI.

[45]  Mark Wallace,et al.  Principles and Practice of Constraint Programming – CP 2004 , 2004, Lecture Notes in Computer Science.

[46]  Nicolas Barnier,et al.  Solving the Kirkman's schoolgirl problem in a few seconds , 2002 .

[47]  Hans Kleine Büning,et al.  Resolution for Quantified Boolean Formulas , 1995, Inf. Comput..

[48]  Joao Marques-Silva,et al.  Theory and Applications of Satisfiability Testing - SAT 2007, 10th International Conference, Lisbon, Portugal, May 28-31, 2007, Proceedings , 2007, SAT.

[49]  Roy Dyckhoff.  Automated Reasoning with Analytic Tableaux and Related Methods , 2000, Lecture Notes in Computer Science.

[50]  Wolfgang Faber,et al.  Logic Programming and Nonmonotonic Reasoning , 2011, Lecture Notes in Computer Science.

[51]  Vasco M. Manquinho,et al.  SAT 2009 competitive events booklet: preliminary version , 2009 .

[52]  Graham Steel,et al.  Deduction with XOR Constraints in Security API Modelling , 2005, CADE.

[53]  Jinbo Huang.  TINISAT in SAT Competition 2008 , 2007 .

[54]  Christian Kern,et al.  Sat7 - Engineering a Modular SAT-Solver , 2006 .

[55]  S. Malik,et al.  Validating the result of a quantified Boolean formula (QBF) solver: theory and practice , 2005, Proceedings of the ASP-DAC 2005. Asia and South Pacific Design Automation Conference, 2005..

[56]  Satoshi Horie,et al.  Hard Instance Generation for SAT (Extended Abstract) , 1997, ISAAC.

[57]  Himanshu Jain,et al.  SAT Solver Descriptions : CMUSAT-Base and CMUSAT , 2007 .

[58]  Osamu Watanabe,et al.  Hard instance generation for SAT , 1998, ArXiv.

[59]  Armando Tacchella,et al.  QuBE++: An Efficient QBF Solver , 2004, FMCAD.

[60]  Albert Oliveras,et al.  The Barcelogic SMT Solver , 2008, CAV.