A Hitchhiker’s Guide to Automatic Differentiation

This article provides an overview of some of the mathematical principles of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator, and explain why they all reduce to the same actual chain of computations.We further give a short mathematical description of some methods of higher-order Forward AD and, at the end of this paper, briefly describe the Reverse Mode of Automatic Differentiation.

[1]  Christian Bischof,et al.  On the Automatic Differentiation of Computer Programs and an Application to Multibody Systems , 1996 .

[2]  Jerzy Karczmarczuk,et al.  Functional Differentiation of Computer Programs , 1998, ICFP '98.

[3]  Andreas Griewank,et al.  Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation , 1992 .

[4]  Barak A. Pearlmutter,et al.  Lazy multivariate higher-order forward-mode AD , 2007, POPL '07.

[5]  Jessika Eichel,et al.  Partial Differential Equations Second Edition , 2016 .

[6]  S. Lang.  Introduction to Differentiable Manifolds , 1964 .

[7]  Amey Karkare,et al.  Heap reference analysis using access graphs , 2006, ACM Trans. Program. Lang. Syst..

[8]  Robert M. Gower,et al.  A new framework for the computation of Hessians , 2012, Optim. Methods Softw..

[9]  Andreas Griewank,et al.  A mathematical view of automatic differentiation , 2003, Acta Numerica.

[10]  Oleksandr Manzyuk.  A Simply Typed λ-Calculus of Forward Automatic Differentiation , 2012, MFPS.

[11]  John H. Hubbard,et al.  A First Look at Differential Algebra , 2011, Am. Math. Mon..

[12]  Laurence C. Dixon.  Automatic Differentiation: Calculation of the Hessian , 2009, Encyclopedia of Optimization.

[13]  G. M.,et al.  Partial Differential Equations I , 2023, Applied Mathematical Sciences.

[14]  Jerzy Karczmarczuk.  Functional Coding of Differential Forms , 1999 .

[15]  Barak A. Pearlmutter,et al.  Nesting forward-mode AD in a functional framework , 2008, High. Order Symb. Comput..

[16]  J. Ritt.  Partial differential algebra , 1950 .

[17]  D. Kalman.  Doubly Recursive Multivariate Automatic Differentiation , 2002 .

[18]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[19]  Barak A. Pearlmutter,et al.  Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator , 2008, TOPL.

[20]  L. B. Rall.  Differentiation and generation of taylor coefficients in PASCAL-SC , 1983 .

[21]  L. B. Rall.  The Arithmetic of Differentiation , 1986 .

[22]  Barak A. Pearlmutter,et al.  DiffSharp: Automatic Differentiation Library , 2015, ArXiv.

[23]  M. Berz.  Differential Algebraic Description of Beam Dynamics to Very High Orders , 1988 .

[25]  R. E. Wengert,et al.  A simple automatic derivative evaluation program , 1964, Commun. ACM.

[26]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.