Stochastic optimization of floating-point programs with tunable precision

The aggressive optimization of floating-point computations is an important problem in high-performance computing. Unfortunately, floating-point instruction sets have complicated semantics that often force compilers to preserve programs as written. We present a method that treats floating-point optimization as a stochastic search problem. We demonstrate the ability to generate reduced precision implementations of Intel's handwritten C numeric library which are up to 6 times faster than the original code, and achieve end-to-end speedups of over 30% on a direct numeric simulation and a ray tracer by optimizing kernels that can tolerate a loss of precision while still remaining correct. Because these optimizations are mostly not amenable to formal verification using the current state of the art, we present a stochastic search technique for characterizing maximum error. The technique comes with an asymptotic guarantee and provides strong evidence of correctness.

[1]  Andy Podgurski,et al.  Reliability, sampling, and algorithmic randomness , 1991, TAV4.

[2]  David Goldberg.  What Every Computer Scientist Should Know About Floating-Point Arithmetic , 1992 .

[3]  J. Geweke,et al.  Priors for Macroeconomic Time Series and Their Application , 1994, Econometric Theory.

[4]  Paul S. Heckbert,et al.  Graphics gems IV , 1994 .

[5]  N.J. Higham,et al.  Numerical Computation: Methods, Software, And Analysis , 1998, IEEE Computational Science and Engineering.

[6]  Douglas Stott Parker,et al.  Monte Carlo arithmetic: how to gamble with floating point and win , 2000, Comput. Sci. Eng..

[7]  Nando de Freitas,et al.  An Introduction to MCMC for Machine Learning , 2004, Machine Learning.

[8]  Sumit Gulwani,et al.  Randomized Algorithms for Program Analysis and Verification , 2005, CAV.

[9]  Sanjit A. Seshia,et al.  Combinatorial sketching for finite programs , 2006, ASPLOS XII.

[10]  Sriram Sankaranarayanan,et al.  State space exploration using feedback constraint generation and Monte-Carlo sampling , 2007, ESEC-FSE '07.

[11]  Fabienne Jézéquel,et al.  CADNA: a library for estimating round-off error propagation , 2008, Comput. Phys. Commun..

[12]  Eric Goubault,et al.  Towards an Industrial Use of FLUCTUAT on Safety-Critical Avionics Software , 2009, FMICS.

[13]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[14]  Sriram Sankaranarayanan,et al.  Numerical stability analysis of floating-point computations using software model checking , 2010, Eighth ACM/IEEE International Conference on Formal Methods and Models for Codesign (MEMOCODE 2010).

[15]  Woongki Baek,et al.  Green: a framework for supporting energy-conscious programming using controlled approximation , 2010, PLDI '10.

[16]  Zhendong Su,et al.  Perturbing numerical calculations for statistical analysis of floating-point program (in)stability , 2010, ISSTA '10.

[17]  Philipp Rümmer,et al.  An SMT-LIB Theory of Binary Floating-Point Arithmetic ∗ , 2010 .

[18]  Daniel M. Roy,et al.  Probabilistically Accurate Program Transformations , 2011, SAS.

[19]  Henry Hoffmann,et al.  Managing performance vs. accuracy trade-offs with loop perforation , 2011, ESEC/FSE '11.

[20]  Paul H. J. Kelly,et al.  Symbolic crosschecking of floating-point and SIMD code , 2011, EuroSys '11.

[21]  Florent de Dinechin,et al.  Certifying the Floating-Point Implementation of an Elementary Function Using Gappa , 2011, IEEE Transactions on Computers.

[22]  Sumit Gulwani,et al.  Synthesis of loop-free programs , 2011, PLDI '11.

[23]  Daniel Kroening,et al.  Deciding floating-point logic with systematic abstraction , 2012, 2012 Formal Methods in Computer-Aided Design (FMCAD).

[24]  Sebastian Hack,et al.  A dynamic program analysis to find floating-point accuracy problems , 2012, PLDI.

[25]  Zeyuan Allen Zhu,et al.  Randomized accuracy-aware program transformations for efficient approximate computations , 2012, POPL '12.

[26]  Sumit Gulwani,et al.  Continuity and robustness of programs , 2012, CACM.

[27]  Bronis R. de Supinski,et al.  Abstract: Automatically Adapting Programs for Mixed-Precision Floating-Point Computation , 2013, 2012 SC Companion: High Performance Computing, Networking Storage and Analysis.

[28]  Emery D. Berger,et al.  STABILIZER: statistically sound performance evaluation , 2013, ASPLOS '13.

[29]  Alexander Aiken,et al.  Data-driven equivalence checking , 2013, OOPSLA.

[30]  Viktor Kuncak,et al.  Sound compilation of reals , 2013, POPL.

[31]  Alexander Aiken,et al.  Stochastic superoptimization , 2012, ASPLOS '13.

[32]  James Demmel,et al.  Precimonious: Tuning assistant for floating-point precision , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[33]  Sofya Raskhodnikova,et al.  Testing and Reconstruction of Lipschitz Functions with Applications to Data Privacy , 2013, SIAM J. Comput..

[34]  Ganesh Gopalakrishnan,et al.  Efficient search for inputs causing high floating-point errors , 2014, PPoPP '14.

[35]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .