The 2016 and 2017 QBF solvers evaluations (QBFEVAL'16 and QBFEVAL'17)

Abstract After a break of about five years, in 2016 the classical QBFEVAL has been revived. QBFEVAL is a competitive evaluation of solvers for quantified Boolean formulas (QBF), the extension of propositional formulas with existential and universal quantifiers over the propositional variables. Due to the enormous interest in QBFEVAL'16, more recently, QBFEVAL'17 was organized. Both competitions were affiliated to the respective editions of the International Conference on Theory and Applications of Satisfiability Testing (SAT'16 and SAT'17), the major conference in research on SAT and related areas. In this paper we report about the 2016 and 2017 competitive evaluations of QBF solvers (QBFEVAL'16 and QBFEVAL'17), the two most recent events in a series of competitions established with the aim of assessing the advancements in reasoning about QBFs. This report gives an overview of the setup of these two events, on their participants and on the results of the experiments that were performed for evaluating the participating systems.

[1]  Mikolás Janota,et al.  Abstraction-Based Algorithm for 2QBF , 2011, SAT.

[2]  Georg Gottlob,et al.  On the computational cost of disjunctive logic programming: Propositional case , 1995, Annals of Mathematics and Artificial Intelligence.

[3]  Enrico Giunchiglia,et al.  sQueezeBF: An Effective Preprocessor for QBFs Based on Equivalence Reasoning , 2010, SAT.

[4]  Stefan Szeider,et al.  Dependency Learning for QBF , 2017, SAT.

[5]  Mikolás Janota,et al.  Solving QBF with Counterexample Guided Refinement , 2012, SAT.

[6]  Armin Biere,et al.  Automated Testing and Debugging of SAT and QBF Solvers , 2010, SAT.

[7]  Christoph Scholl,et al.  The QBF Solver AIGSolve , 2016, QBF@SAT.

[8]  Armin Biere,et al.  DepQBF: A Dependency-Aware QBF Solver , 2010, J. Satisf. Boolean Model. Comput..

[9]  Shahab Tasharrofi,et al.  Solving QBF Instances with Nested SAT Solvers , 2016, AAAI Workshop: Beyond NP.

[10]  Markus N. Rabe,et al.  CAQE: A Certifying QBF Solver , 2015, 2015 Formal Methods in Computer-Aided Design (FMCAD).

[11]  Konstantin Korovin,et al.  Inst-Gen - A Modular Approach to Instantiation-Based Automated Reasoning , 2013, Programming Logics.

[12]  Bart Selman,et al.  Boosting Combinatorial Search Through Randomization , 1998, AAAI/IAAI.

[13]  Reinhold Letz,et al.  Lemma and Model Caching in Decision Procedures for Quantified Boolean Formulas , 2002, TABLEAUX.

[14]  Luca Pulina,et al.  The QBFEVAL Web Portal , 2006, JELIA.

[15]  Allen Van Gelder.  Primal and Dual Encoding from Applications into Quantified Boolean Formulas , 2013, CP.

[16]  Marijn J. H. Heule,et al.  SAT Competition 2016: Recent Developments , 2017, AAAI.

[17]  Armin Biere,et al.  Failed Literal Detection for QBF , 2011, SAT.

[18]  Leander Tentrup.  Non-prenex QBF Solving Using Abstraction , 2016, SAT.

[19]  Lukasz Kaiser,et al.  MPIDepQBF: Towards Parallel QBF Solving without Knowledge Sharing , 2014, SAT.

[20]  Sharad Malik,et al.  Towards a Symmetric Treatment of Satisfaction and Conflicts in Quantified Boolean Formula Evaluation , 2002, CP.

[21]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[22]  Edmund M. Clarke,et al.  A Non-prenex, Non-clausal QBF Solver with Game-State Learning , 2010, SAT.

[23]  Adnan Darwiche,et al.  A Lightweight Component Caching Scheme for Satisfiability Solvers , 2007, SAT.

[24]  Sharad Malik,et al.  Efficient conflict driven learning in a Boolean satisfiability solver , 2001, IEEE/ACM International Conference on Computer Aided Design. ICCAD 2001. IEEE/ACM Digest of Technical Papers (Cat. No.01CH37281).

[25]  Luca Pulina.  The Ninth QBF Solvers Evaluation - Preliminary Report , 2016, QBF@SAT.

[26]  Armin Biere,et al.  SAT Race 2015 , 2016, Artif. Intell..

[27]  Luca Pulina,et al.  A self-adaptive multi-engine solver for quantified Boolean formulas , 2009, Constraints.

[28]  Florian Lonsing,et al.  HordeQBF: A Modular and Massively Parallel QBF Solver , 2016, SAT.

[29]  Armin Biere,et al.  qbf2epr: A Tool for Generating EPR Formulas from QBF , 2013, PAAR@IJCAR.

[30]  Roderick Bloem,et al.  QBF Solving by Counterexample-guided Expansion , 2016, ArXiv.

[31]  Stefan Szeider,et al.  Soundness of Q-resolution with dependency schemes , 2016, Theor. Comput. Sci..

[32]  Yannet Interian,et al.  A Model for Generating Random Quantified Boolean Formulas , 2005, IJCAI.

[33]  Miroslaw Truszczynski,et al.  Generating Hard Random Boolean Formulas and Disjunctive Logic Programs , 2017, IJCAI.

[34]  Luca Pulina,et al.  Evaluating and certifying QBFs: A comparison of state-of-the-art tools , 2009, AI Commun..

[35]  Mikolás Janota,et al.  The QBFGallery 2014: The QBF Competition at the FLoC Olympic Games , 2014, J. Satisf. Boolean Model. Comput..

[36]  Peter Sanders,et al.  HordeSat: A Massively Parallel Portfolio SAT Solver , 2015, SAT.

[37]  Florian Lonsing,et al.  DepQBF 6.0: A Search-Based QBF Solver Beyond Traditional QCDCL , 2017, CADE.

[38]  Mikolás Janota,et al.  Solving QBF by Clause Selection , 2015, IJCAI.

[39]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[40]  Christoph Scholl,et al.  Exploiting structure in an AIG based QBF solver , 2009, 2009 Design, Automation & Test in Europe Conference & Exhibition.

[41]  David Déharbe,et al.  The 2014 SMT Competition , 2014, J. Satisf. Boolean Model. Comput..

[42]  Christoph Scholl,et al.  An AIG-based QBF-solver using SAT for preprocessing , 2010, Design Automation Conference.

[43]  Armando Tacchella,et al.  Watched Data Structures for QBF Solvers , 2003, SAT.

[44]  Cesare Tinelli,et al.  Introducing StarExec: a Cross-Community Infrastructure for Logic Solving , 2014, COMPARE.

[45]  Inês Lynce,et al.  The Seventh QBF Solvers Evaluation (QBFEVAL'10) , 2010, SAT.

[46]  Luca Pulina,et al.  A Structural Approach to Reasoning with Quantified Boolean Formulas , 2009, IJCAI.

[47]  Armin Biere,et al.  Blocked Clause Elimination for QBF , 2011, CADE.

[48]  Armin Biere,et al.  Resolution-Based Certificate Extraction for QBF - (Tool Presentation) , 2012, SAT.

[49]  Sharad Malik,et al.  Conflict driven learning in a quantified Boolean Satisfiability solver , 2002, ICCAD 2002.

[50]  Luca Pulina,et al.  Twelve Years of QBF Evaluations: QSAT Is PSPACE-Hard and It Shows , 2016, Fundam. Informaticae.

[51]  Sanjit A. Seshia,et al.  Incremental Determinization , 2016, SAT.

[52]  Bernd Becker,et al.  HQSpre - An Effective Preprocessor for QBF and DQBF , 2017, TACAS.

[53]  Armin Biere,et al.  Enhancing Search-Based QBF Solving by Dynamic Blocked Clause Elimination , 2015, LPAR.

[54]  Jie-Hong Roland Jiang,et al.  QELL: QBF Reasoning with Extended Clause Learning and Levelized SAT Solving , 2015, SAT.