Just fuzz it: solving floating-point constraints using coverage-guided fuzzing

We investigate the use of coverage-guided fuzzing as a means of proving satisfiability of SMT formulas over finite variable domains, with specific application to floating-point constraints. We show how an SMT formula can be encoded as a program containing a location that is reachable if and only if the program’s input corresponds to a satisfying assignment to the formula. A coverage-guided fuzzer can then be used to search for an input that reaches the location, yielding a satisfying assignment. We have implemented this idea in a tool, Just Fuzz-it Solver (JFS), and we present a large experimental evaluation showing that JFS is both competitive with and complementary to state-of-the-art SMT solvers with respect to solving floating-point constraints, and that the coverage-guided approach of JFS provides significant benefit over naive fuzzing in the floating-point domain. Applied in a portfolio manner, the JFS approach thus has the potential to complement traditional SMT solvers for program analysis tasks that involve reasoning about floating-point constraints.

[1]  James Demmel,et al.  Floating-Point Precision Tuning Using Blame Analysis , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[2]  Arnaud Gotlieb,et al.  Symbolic Path-Oriented Test Data Generation for Floating-Point Programs , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[3]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[4]  Sigal Asaf,et al.  FPgen - a test generation framework for datapath floating-point verification , 2003, Eighth IEEE International High-Level Design Validation and Test Workshop.

[5]  Miriam Leeser,et al.  Behavioral Non-portability in Scientific Numeric Computing , 2015, Euro-Par.

[6]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[7]  Corina S. Pasareanu,et al.  Symbolic Execution for Checking the Accuracy of Floating-Point Programs , 2015, SOEN.

[8]  Sumit Gulwani,et al.  Program Synthesis , 2017, Software Systems Safety.

[9]  Yahia Lebbah,et al.  Solving Constraints over Floating-Point Numbers , 2001, CP.

[10]  Alberto Griggio,et al.  The MathSAT5 SMT Solver , 2013, TACAS.

[11]  Zhendong Su,et al.  Achieving high coverage for floating-point code via unconstrained programming , 2017, PLDI.

[12]  Clark W. Barrett,et al.  The SMT-LIB Standard Version 2.0 , 2010 .

[13]  Daniel Kroening,et al.  Hardware verification using ANSI-C programs as a reference , 2003, ASP-DAC '03.

[14]  R. Weisberg A-N-D , 2011 .

[15]  François Bobot,et al.  Real Behavior of Floating Point , 2017, SMT.

[16]  Miriam Leeser,et al.  Make it real: Effective floating-point reasoning via exact arithmetic , 2014, 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[17]  Olivier Ponsini,et al.  Generating test cases inside suspicious intervals for floating-point number programs , 2014, CSTVA 2014.

[18]  James Demmel,et al.  Precimonious: Tuning assistant for floating-point precision , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[19]  Zhendong Su,et al.  Automatic detection of floating-point exceptions , 2013, POPL.

[20]  Klaus Wehrle,et al.  Floating-point symbolic execution: A case study in N-version programming , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[21]  Nikolaj Bjørner,et al.  Satisfiability modulo theories , 2011, Commun. ACM.

[22]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[23]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[24]  Paul H. J. Kelly,et al.  Symbolic Testing of OpenCL Code , 2011, Haifa Verification Conference.

[25]  Patrick Cousot,et al.  A static analyzer for large safety-critical software , 2003, PLDI '03.

[26]  Marcelo d'Amorim,et al.  Randomized constraint solvers: a comparative study , 2010, Innovations in Systems and Software Engineering.

[27]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[28]  Dominik Stoffel,et al.  goSAT: Floating-point satisfiability as global optimization , 2017, 2017 Formal Methods in Computer Aided Design (FMCAD).

[29]  Paul H. J. Kelly,et al.  Symbolic crosschecking of floating-point and SIMD code , 2011, EuroSys '11.

[30]  G. S. Tseitin On the Complexity of Derivation in Propositional Calculus , 1983 .

[31]  Marcelo d'Amorim,et al.  Symbolic Execution with Interval Solving and Meta-heuristic Search , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[32]  Arnaud Gotlieb,et al.  Symbolic execution of floating‐point computations , 2006, Softw. Test. Verification Reliab..

[33]  Corina S. Pasareanu,et al.  Symbolic PathFinder: integrating symbolic execution with model checking for Java bytecode analysis , 2013, Automated Software Engineering.

[34]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[35]  Laurence Rideau,et al.  A Generic Library for Floating-Point Numbers and Its Application to Exact Computing , 2001, TPHOLs.

[36]  Armando Solar-Lezama,et al.  Towards optimization-safe systems: analyzing the impact of undefined behavior , 2013, SOSP.

[37]  Marcelo d'Amorim,et al.  CORAL: Solving Complex Constraints for Symbolic PathFinder , 2011, NASA Formal Methods.

[38]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[39]  Cesare Tinelli,et al.  CVC4 at the SMT Competition 2018 , 2018, ArXiv.

[40]  Eric Goubault,et al.  Static Analysis-Based Validation of Floating-Point Computations , 2003, Numerical Software with Result Verification.

[41]  Minghui Quan Hotspot symbolic execution of floating-point programs , 2016, SIGSOFT FSE.

[42]  Santosh Nagarakatte,et al.  Alive-FP: Automated Verification of Floating Point Based Peephole Optimizations in LLVM , 2016, SAS.

[43]  Cristian Cadar,et al.  Multi-solver Support in Symbolic Execution , 2013, SMT.

[44]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[45]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[46]  Jan Peleska,et al.  Automated Test Case Generation with SMT-Solving and Abstract Interpretation , 2011, NASA Formal Methods.

[47]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[48]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[49]  Sumit Gulwani,et al.  Synthesis of loop-free programs , 2011, PLDI '11.

[50]  Pavel Panchekha,et al.  Automatically improving accuracy for floating point expressions , 2015, PLDI.

[51]  Andres Nötzli,et al.  LifeJacket: verifying precise floating-point optimizations in LLVM , 2016, SOAP@PLDI.

[52]  Zhendong Su,et al.  XSat: A Fast Floating-Point Satisfiability Solver , 2016, CAV.

[53]  Ian Briggs,et al.  Rigorous floating-point mixed-precision tuning , 2017, POPL.

[54]  Wen-Chuan Lee,et al.  RAIVE: runtime assessment of floating-point instability by vectorization , 2015, OOPSLA 2015.

[55]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[56]  K. Rustan M. Leino,et al.  Dafny: An Automatic Program Verifier for Functional Correctness , 2010, LPAR.