Toward a Standard Benchmark Format and Suite for Floating-Point Analysis

We introduce FPBench, a standard benchmark format for validation and optimization of numerical accuracy in floating-point computations. FPBench is a first step toward addressing an increasing need in our community for comparisons and combinations of tools from different application domains. To this end, FPBench provides a basic floating-point benchmark format and accuracy measures for comparing different tools. The FPBench format and measures allow comparing and composing different floating-point tools. We describe the FPBench format and measures and show that FPBench expresses benchmarks from recent papers in the literature, by building an initial benchmark suite drawn from these papers. We intend for FPBench to grow into a standard benchmark suite for the members of the floating-point tools research community.

[1]  Patrick Cousot,et al.  Abstract Interpretation Frameworks , 1992, J. Log. Comput..

[2]  Richard W. Hamming,et al.  Numerical methods for scientists and engineers (2nd ed.) , 1986 .

[3]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[4]  Matthieu Martel.  An Overview of Semantics for the Validation of Numerical Programs , 2005, VMCAI.

[5]  Eric Goubault,et al.  Static Analysis of Finite Precision Computations , 2011, VMCAI.

[6]  Assalé Adjé,et al.  Quadratic Zonotopes - An Extension of Zonotopes to Quadratic Arithmetics , 2014, APLAS.

[7]  Viktor Kuncak,et al.  Sound compilation of reals , 2013, POPL.

[8]  Sriram Sankaranarayanan.  Static Analysis in the Continuously Changing World , 2013, SAS.

[9]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[10]  Matthieu Martel,et al.  Some future challenges in the validation of control systems , 2006 .

[11]  Pavel Panchekha,et al.  Automatically improving accuracy for floating point expressions , 2015, PLDI.

[12]  Alexander Aiken,et al.  Verifying bit-manipulations of floating-point , 2016, PLDI.

[13]  Patrick Cousot,et al.  Static Analysis and Verification of Aerospace Software by Abstract Interpretation , 2010, Found. Trends Program. Lang..

[14]  Eric Goubault,et al.  Static Analysis by Abstract Interpretation of Numerical Programs and Systems, and FLUCTUAT , 2013, SAS.

[15]  Gerard J. Holzmann,et al.  Mars code , 2014, CACM.

[16]  Clark W. Barrett,et al.  The SMT-LIB Standard Version 2.0 , 2010 .

[17]  Matthieu Martel,et al.  Intra-procedural Optimization of the Numerical Accuracy of Programs , 2015, FMICS.

[18]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[19]  Dawson R. Engler,et al.  A few billion lines of code later , 2010, Commun. ACM.

[20]  John L. Henning.  SPEC CPU2006 benchmark descriptions , 2006, CARN.

[21]  Alexander Aiken,et al.  Stochastic optimization of floating-point programs with tunable precision , 2014, PLDI.

[22]  Nikolaj Bjørner,et al.  FM 2015 : formal methods : 20th International Symposium, Oslo, Norway, June 24-26, 2015 : proceedings , 2015 .

[23]  Ganesh Gopalakrishnan,et al.  Rigorous Estimation of Floating-Point Round-off Errors with Symbolic Taylor Expansions , 2015, FM.

[24]  Anoop Gupta,et al.  The SPLASH-2 programs: characterization and methodological considerations , 1995, ISCA.

[25]  Alexandre Chapoutot,et al.  Interval Slopes as a Numerical Abstract Domain for Floating-Point Variables , 2010, SAS.