MedleySolver: Online SMT Algorithm Selection

Satisfiability modulo theories (SMT) solvers implement a wide range of optimizations that are often tailored to a particular class of problems, and that differ significantly between solvers. As a result, one solver may solve a query quickly while another might be flummoxed completely. Predicting the performance of a given solver is difficult for users of SMT-driven applications, particularly when the problems they have to solve do not fall neatly into a well-understood category. In this paper, we propose an online algorithm selection framework for SMT called MedleySolver that predicts the relative performances of a set of SMT solvers on a given query, distributes time amongst the solvers, and deploys the solvers in sequence until a solution is obtained. We evaluate MedleySolver against the best available alternative, an offline learning technique, in terms of pure performance and practical usability for a typical SMT user. We find that with no prior training, MedleySolver solves 93.9% of the queries solved by the virtual best solver selector achieving 59.8% of the par-2 score of the most successful individual solver, which solves 87.3%. For comparison, the best available alternative takes longer to train than MedleySolver takes to solve our entire set of 2000 queries.

[1]  Roberto Bruttomesso,et al.  The MathSAT 4SMT Solver , 2008, CAV.

[2]  Sanjit A. Seshia,et al.  Modeling and Verifying Systems Using a Logic of Counter Arithmetic with Lambda Expressions and Uninterpreted Functions , 2002, CAV.

[3]  Sanjit A. Seshia,et al.  Beaver: Engineering an Efficient SMT Solver for Bit-Vector Arithmetic , 2009, CAV.

[4]  Sanjit A. Seshia,et al.  Combinatorial sketching for finite programs , 2006, ASPLOS XII.

[5]  Christophe Lecoutre,et al.  Learning Variable Ordering Heuristics with Multi-Armed Bandits and Restarts , 2020, ECAI.

[6]  Peter Auer,et al.  The Nonstochastic Multiarmed Bandit Problem , 2002, SIAM J. Comput..

[7]  Predrag Janicic,et al.  Simple algorithm portfolio for SAT , 2011, Artificial Intelligence Review.

[8]  Michèle Sebag,et al.  BESS: Bandit Ensemble for parallel SAT Solving , 2012 .

[9]  Sanjit A. Seshia,et al.  Adaptive eager boolean encoding for arithmetic reasoning in verification , 2005 .

[10]  Pauline Bolignano,et al.  Semantic-based Automated Reasoning for AWS Access Policies using SMT , 2018, 2018 Formal Methods in Computer Aided Design (FMCAD).

[11]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[12]  James F. Power,et al.  Predicting SMT Solver Performance for Software Verification , 2017, F-IDE@FM.

[13]  Jan Strejcek,et al.  Solving Quantified Bit-Vector Formulas Using Binary Decision Diagrams , 2016, SAT.

[14]  Mislav Balunovic,et al.  Learning to Solve SMT Formulas , 2018, NeurIPS.

[15]  Souheib Baarir,et al.  Parallel Learning Portfolio-based solvers , 2017, ICCS.

[16]  Roland H. C. Yap,et al.  Learning Robust Search Strategies Using a Bandit-Based Approach , 2018, AAAI.

[17]  Aina Niemetz,et al.  Bitwuzla at the SMT-COMP 2020 , 2020, ArXiv.

[18]  Christopher L. Conway,et al.  Cvc4 , 2011, CAV.

[19]  Quanquan Gu,et al.  Neural Contextual Bandits with Upper Confidence Bound-Based Exploration , 2019, ArXiv.

[20]  Nikolaj Bjørner,et al.  SMT Solvers for Testing, Program Analysis and Verification at Microsoft , 2009, 2009 11th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing.

[21]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[22]  Andrei Voronkov,et al.  Coming to terms with quantified reasoning , 2016, POPL.

[23]  Eoin O'Mahony,et al.  Using Case-based Reasoning in an Algorithm Portfolio for Constraint Solving ? , 2008 .

[24]  Daniel Kroening,et al.  Decision Procedures - An Algorithmic Point of View , 2008, Texts in Theoretical Computer Science. An EATCS Series.

[25]  Leonardo Mendonça de Moura,et al.  Complete Instantiation for Quantified Formulas in Satisfiabiliby Modulo Theories , 2009, CAV.

[26]  I. J. Myung,et al.  Tutorial on maximum likelihood estimation , 2003 .

[27]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[28]  Sanjit A. Seshia,et al.  UCLID5: Integrating Modeling, Verification, Synthesis and Learning , 2018, 2018 16th ACM/IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE).

[29]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[30]  Sylvain Conchon,et al.  The SMT Competition 2015-2018 , 2019, J. Satisf. Boolean Model. Comput..

[31]  Youssef Hamadi,et al.  A Concurrent Portfolio Approach to SMT Solving , 2009, CAV.

[32]  Armin Biere,et al.  Boolector 2.0 , 2015, J. Satisf. Boolean Model. Comput..

[33]  Bruno Dutertre,et al.  Yices 2.2 , 2014, CAV.

[34]  Trevor Hansen,et al.  A constraint solver and its application to machine code test generation , 2012 .

[35]  Christof Löding,et al.  Foundations for natural proofs and quantifier instantiation , 2017, Proc. ACM Program. Lang..

[36]  Joël Ouaknine,et al.  Deciding Bit-Vector Arithmetic with Abstraction , 2007, TACAS.

[37]  Alan J. Hu,et al.  Boosting Verification by Automatic Tuning of Decision Procedures , 2007, Formal Methods in Computer Aided Design (FMCAD'07).

[38]  Shipra Agrawal,et al.  Analysis of Thompson Sampling for the Multi-armed Bandit Problem , 2011, COLT.

[39]  Aina Niemetz,et al.  MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers , 2021, TACAS.

[40]  Wei Chu,et al.  A contextual-bandit approach to personalized news article recommendation , 2010, WWW '10.

[41]  Enrique F. Castillo,et al.  Learning and Updating of Uncertainty in Dirichlet Models , 2004, Machine Learning.

[42]  Shipra Agrawal,et al.  Thompson Sampling for Contextual Bandits with Linear Payoffs , 2012, ICML.

[43]  Viktor Kuncak,et al.  Counterexample-Guided Quantifier Instantiation for Synthesis in SMT , 2015, CAV.