Floating-point symbolic execution: A case study in N-version programming

Symbolic execution is a well-known program analysis technique for testing software, which makes intensive use of constraint solvers. Recent support for floating-point constraint solving has made it feasible to support floating-point reasoning in symbolic execution tools. In this paper, we present the experience of two research teams that independently added floating-point support to KLEE, a popular symbolic execution engine. Since the two teams independently developed their extensions, this created the rare opportunity to conduct a rigorous comparison between the two implementations, essentially a modern case study on N-version programming. As part of our comparison, we report on the different design and implementation decisions taken by each team, and show their impact on a rigorously assembled and tested set of benchmarks, itself a contribution of the paper.

[1]  Minghui Quan.  Hotspot symbolic execution of floating-point programs , 2016, SIGSOFT FSE.

[2]  Daniel Kroening,et al.  Deciding floating-point logic with abstract conflict driven clause learning , 2013, Formal Methods in System Design.

[3]  Algirdas Avizienis,et al.  The N-Version Approach to Fault-Tolerant Software , 1985, IEEE Transactions on Software Engineering.

[4]  Guillaume Melquiond,et al.  Combining Coq and Gappa for Certifying Floating-Point Programs , 2009, Calculemus/MKM.

[5]  Bjarne Stroustrup,et al.  C++ Programming Language , 1986, IEEE Softw..

[6]  Daniel Kroening,et al.  Mixed abstractions for floating-point arithmetic , 2009, 2009 Formal Methods in Computer-Aided Design.

[7]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[8]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[9]  Liming Chen,et al.  N-VERSION PROGRAMMINC: A FAULT-TOLERANCE APPROACH TO RELlABlLlTY OF SOFTWARE OPERATlON , 1995, Twenty-Fifth International Symposium on Fault-Tolerant Computing, 1995, ' Highlights from Twenty-Five Years'..

[10]  Cesare Tinelli,et al.  Solving SAT and SAT Modulo Theories: From an abstract Davis--Putnam--Logemann--Loveland procedure to DPLL(T) , 2006, JACM.

[11]  Cesare Tinelli,et al.  An Automatable Formal Semantics for IEEE-754 Floating-Point Arithmetic , 2015, 2015 IEEE 22nd Symposium on Computer Arithmetic.

[12]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[13]  Philipp Rümmer,et al.  An SMT-LIB Theory of Binary Floating-Point Arithmetic ∗ , 2010 .

[14]  Daniel Kroening,et al.  Interpolation-Based Verification of Floating-Point Programs with Abstract CDCL , 2013, SAS.

[15]  Arnaud Gotlieb,et al.  Symbolic execution of floating‐point computations , 2006, Softw. Test. Verification Reliab..

[16]  George A. Constantinides,et al.  Certified Roundoff Error Bounds Using Semidefinite Programming , 2015, ACM Trans. Math. Softw..

[17]  Corina S. Pasareanu,et al.  Symbolic PathFinder: integrating symbolic execution with model checking for Java bytecode analysis , 2013, Automated Software Engineering.

[18]  Santosh Nagarakatte,et al.  Alive-FP: Automated Verification of Floating Point Based Peephole Optimizations in LLVM , 2016, SAS.

[19]  Marcelo d'Amorim,et al.  Symbolic Execution with Interval Solving and Meta-heuristic Search , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[20]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[21]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[22]  Ganesh Gopalakrishnan,et al.  Rigorous Estimation of Floating-Point Round-off Errors with Symbolic Taylor Expansions , 2015, FM.

[23]  Corina S. Pasareanu,et al.  Symbolic Execution for Checking the Accuracy of Floating-Point Programs , 2015, SOEN.

[24]  SenKoushik,et al.  Symbolic execution for software testing , 2013 .

[25]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[26]  David J. Harper,et al.  Paranoia , 2009, The Harvard mental health letter.

[27]  Philipp Rümmer,et al.  Approximations for Model Construction , 2014, IJCAR.

[28]  Eric Goubault,et al.  Static Analysis-Based Validation of Floating-Point Computations , 2003, Numerical Software with Result Verification.

[29]  Zhendong Su,et al.  Automatic detection of floating-point exceptions , 2013, POPL.

[30]  Parosh Aziz Abdulla.  Behavioral Non-Portability in Scientific Numeric Computing , 2015 .

[31]  Marcelo d'Amorim,et al.  CORAL: Solving Complex Constraints for Symbolic PathFinder , 2011, NASA Formal Methods.

[32]  Jean-Christophe Filliâtre,et al.  Formal Verification of Floating-Point Programs , 2007, 18th IEEE Symposium on Computer Arithmetic (ARITH '07).

[33]  Viktor Kuncak,et al.  Sound compilation of reals , 2013, POPL.

[34]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[35]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[36]  Arnaud Gotlieb,et al.  Symbolic Path-Oriented Test Data Generation for Floating-Point Programs , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[37]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[38]  Arnaud Gotlieb,et al.  Symbolic execution of floating-point computations: Research Articles , 2006 .

[39]  Eric Goubault,et al.  Asserting the Precision of Floating-Point Computations: A Simple Abstract Interpreter , 2002, ESOP.

[40]  Miriam Leeser,et al.  Make it real: Effective floating-point reasoning via exact arithmetic , 2014, 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[41]  Olivier Ponsini,et al.  Verifying floating-point programs with constraint programming and abstract interpretation techniques , 2014, Automated Software Engineering.

[42]  Olivier Ponsini,et al.  Generating test cases inside suspicious intervals for floating-point number programs , 2014, CSTVA 2014.

[43]  Philipp Rümmer,et al.  An Approximation Framework for Solvers and Decision Procedures , 2016, Journal of Automated Reasoning.

[44]  Daniel Kroening,et al.  Deciding floating-point logic with systematic abstraction , 2012, 2012 Formal Methods in Computer-Aided Design (FMCAD).

[45]  Hélène Collavizza,et al.  Searching Critical Values for Floating-Point Programs , 2016, ICTSS.

[46]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[47]  James Demmel,et al.  IEEE Standard for Floating-Point Arithmetic , 2008 .

[48]  Paul H. J. Kelly,et al.  Symbolic crosschecking of floating-point and SIMD code , 2011, EuroSys '11.

[49]  Derek Bruening,et al.  AddressSanitizer: A Fast Address Sanity Checker , 2012, USENIX Annual Technical Conference.

[50]  Olivier Ponsini,et al.  Combining Constraint Programming and Abstract Interpretation for Value Analysis of Floating-point Programs , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[51]  Sigal Asaf,et al.  FPgen - a test generation framework for datapath floating-point verification , 2003, Eighth IEEE International High-Level Design Validation and Test Workshop.

[52]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[53]  Dirk Merkel,et al.  Docker: lightweight Linux containers for consistent development and deployment , 2014 .

[54]  Anthony Di Franco,et al.  A comprehensive study of real-world numerical bug characteristics , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[55]  Eric Goubault,et al.  Static Analyses of the Precision of Floating-Point Operations , 2001, SAS.

[56]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[57]  Daniel Kroening,et al.  Decision Procedures - An Algorithmic Point of View , 2008, Texts in Theoretical Computer Science. An EATCS Series.

[58]  Paul H. J. Kelly,et al.  Symbolic Testing of OpenCL Code , 2011, Haifa Verification Conference.