On the numerical stability of algorithmic differentiation

In contrast to integration, the differentiation of a function is an ill-conditioned process, if only an oracle is available for its pointwise evaluation. That is, unrelated small variations in the value of the composite function are allowed at nearly identical arguments. In contrast, we show here that, if the function is defined by an evaluation procedure as a composition of arithmetic operations and elementary functions, then automatic, or algorithmic differentiation is backward stable in the sense of Wilkinson. More specifically, the derivative values obtained are exact for a perturbation of the elementary components at the level of the machine precision. We also provide a forward error analysis for both the forward and reverse mode. The theoretical analysis is confirmed by numerical experiments.

[1]  Andreas Griewank,et al.  Automatic Differentiation of Algorithms: From Simulation to Optimization , 2000, Springer New York.

[2]  Andreas Griewank,et al.  Evaluating higher derivative tensors by forward propagation of univariate Taylor series , 2000, Math. Comput..

[3]  Nicholas J. Higham,et al.  INVERSE PROBLEMS NEWSLETTER , 1991 .

[4]  Mei Han An,et al.  accuracy and stability of numerical algorithms , 1991 .

[5]  Siegfried M. Rump,et al.  Least significant bit evaluation of arithmetic expressions in single-precision , 1983, Computing.

[6]  Jean-Michel Muller,et al.  Handbook of Floating-Point Arithmetic (2nd Ed.) , 2018 .

[7]  Webb Miller,et al.  Software for Roundoff Analysis of Matrix Algorithms , 1980 .

[8]  S. Rump.  Rigorous and Portable Standard Functions , 2001 .

[9]  James Demmel,et al.  IEEE Standard for Floating-Point Arithmetic , 2008 .

[10]  A. Griewank,et al.  Automatic differentiation of algorithms : theory, implementation, and application , 1994 .

[11]  Martin Berz,et al.  Computational differentiation : techniques, applications, and tools , 1996 .

[12]  Fermín S. V. Bazán,et al.  Matrix polynomials with partially prescribed eigenstructure: eigenvalue sensitivity and condition estimation , 2005 .

[13]  Michael C. Ring.  MAPM, a portable arbitrary precision math library in C , 2001 .

[14]  Siegfried M. Rump,et al.  Accurate Sum and Dot Product , 2005, SIAM J. Sci. Comput..

[15]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[16]  M. Michelsen.  The isothermal flash problem. Part I. Stability , 1982 .

[17]  E. Oblow,et al.  Sensitivity Theory for General Systems of Nonlinear Equations , 1980 .

[18]  Robert Kozma,et al.  Investigation of stochastic reactor noise models--A one-variable space-time-dependent model , 1984 .

[19]  James Demmel,et al.  Design, implementation and testing of extended and mixed precision BLAS , 2000, TOMS.

[20]  Abraham Ziv,et al.  Fast evaluation of elementary mathematical functions with correctly rounded last bit , 1991, TOMS.

[21]  S. Linnainmaa.  Taylor expansion of the accumulated rounding error , 1976 .

[22]  T. Tsuchiya,et al.  Automatic computation of partial derivatives and rounding error estimates with applications to large-scale systems of nonlinear equations , 1988 .

[23]  B. Speelpenning.  Compiling Fast Partial Derivatives of Functions Given by Algorithms , 1980 .

[24]  Bruce Christianson,et al.  Reverse accumulation and accurate rounding error estimates for taylor series , 1992 .

[25]  Uwe Naumann,et al.  Automatic Differentiation: Applications, Theory, and Implementations (Lecture Notes in Computational Science and Engineering) , 2006 .

[26]  J. H. Wilkinson.  Modern Error Analysis , 1971 .

[27]  Arnaud Tisserand,et al.  Toward Correctly Rounded Transcendentals , 1998, IEEE Trans. Computers.

[28]  Richard J. Fateman,et al.  Automatic Differentiation of Algorithms: Theory, Implementation, and Application (Andreas Griewank and George F. Corliss, eds.) , 1993, SIAM Rev..

[29]  Martin Bücker,et al.  Automatic differentiation : applications, theory, and implementations , 2006 .