BanditFuzz: A Reinforcement-Learning Based Performance Fuzzer for SMT Solvers

Satisfiability Modulo Theories (SMT) solvers are fundamental tools that are used widely in software engineering, verification, and security research. Precisely because of their widespread use, it is imperative we develop efficient and systematic methods to test them. To this end, we present a reinforcement-learning based fuzzing system, BanditFuzz, that learns grammatical constructs of well-formed inputs that may cause performance slowdown in SMT solvers. To the best of our knowledge, BanditFuzz is the first machine-learning based performance fuzzer for SMT solvers. BanditFuzz takes the following as input: a grammar G describing wellformed inputs to a set of distinct solvers (say, a target solver T and a reference solver R) that implement the same specification, and a fuzzing objective (e.g., aim to maximize the relative performance difference between T and R). BanditFuzz outputs a list of grammatical constructs that are ranked in descending order by how likely they are to increase the performance difference between solvers T and R. Using BanditFuzz, we constructed two benchmark suites (with 400 floating-point and 300 string instances) that expose performance issues in all considered solvers, namely, Z3, CVC4, Colibri, MathSAT, Z3seq, and Z3str3. We also performed a comparison of BanditFuzz against random, mutation, and evolutionary fuzzing methods and observed up to a 81% improvement based on PAR-2 scores used in SAT competitions. That is, relative to other fuzzing methods considered, BanditFuzz was found to be more efficient at constructing inputs with wider performance margin between a target and a set of reference solvers.