On computational differentiation

Numerical derivatives play an important role in many computations. In many applications, the cost associated with evaluation of numerical derivatives may be significant. Dramatic improvements in the speed of such calculations can be obtained through careful consideration of how these derivatives are computed. This paper reviews several ways in which numerical derivatives can be evaluated: hand-coding, finite difference approximations, reverse polish notation evaluation, symbolic differentiation, and automatic differentiation. It is concluded that automatic differentiation has significant advantages over all other approaches. Several ways of improving the efficiency of obtaining derivatives in an interpretive, symbolic environment are discussed. Example problems are compared to illustrate these improvements.

[1]  Andreas Griewank,et al.  Algorithm 755: ADOL-C: a package for the automatic differentiation of algorithms written in C/C++ , 1996, TOMS.

[2]  Paul I. Barton,et al.  Modeling of combined discrete/continuous processes , 1994 .

[3]  B. Speelpenning.  Compiling Fast Partial Derivatives of Functions Given by Algorithms , 1980 .

[4]  E. H. Chimowitz,et al.  An Equation Analyzer package for the manipulation of mathematical expressions—I , 1981 .

[5]  K. Tanabe,et al.  Mathematical Programming: Recent Developments and Applications. , 1990 .

[6]  Mark A. Kramer,et al.  Sensitivity analysis of systems of differential and algebraic equations , 1985 .

[7]  M. Powell,et al.  On the Estimation of Sparse Jacobian Matrices , 1974 .

[8]  Paul I. Barton,et al.  Efficient Calculation of Sparse Jacobians , 1999, SIAM J. Sci. Comput..

[9]  Martin Berz,et al.  Computational differentiation : techniques, applications, and tools , 1996 .

[10]  A. Griewank,et al.  On the calculation of Jacobian matrices by the Markowitz rule , 1991 .

[11]  Bruce Christianson,et al.  Automatic Hessians by reverse accumulation , 1992 .

[12]  Andreas Griewank,et al.  ADIFOR - Generating Derivative Codes form Fortran Programs , 1992, Sci. Program..

[13]  Louis B. Rall,et al.  Automatic Differentiation: Techniques and Applications , 1981, Lecture Notes in Computer Science.

[14]  L. Petzold,et al.  Numerical methods and software for sensitivity analysis of differential-algebraic systems , 1986 .

[15]  Paul S. Wang,et al.  MACSYMA from F to G , 1985, J. Symb. Comput..

[16]  J. K. Reid,et al.  MA48: A FORTRAN code for direct solution of sparse unsymmetric linear systems of equations , 1993 .

[17]  J. W. Ponton.  The numerical evaluation of analytical derivatives , 1982 .

[18]  T. Tsuchiya,et al.  Automatic computation of partial derivatives and rounding error estimates with applications to large-scale systems of nonlinear equations , 1988 .

[19]  Linda R. Petzold,et al.  Numerical solution of initial-value problems in differential-algebraic equations , 1996, Classics in applied mathematics.

[20]  A. Griewank,et al.  Automatic differentiation of algorithms : theory, implementation, and application , 1994 .

[21]  P. I. Barton,et al.  Efficient sensitivity analysis of large-scale differential-algebraic systems , 1997 .