Beautiful differentiation

Automatic differentiation (AD) is a precise, efficient, and convenient method for computing derivatives of functions. Its forward-mode implementation can be quite simple even when extended to compute all of the higher-order derivatives as well. The higher-dimensional case has also been tackled, though with extra complexity. This paper develops an implementation of higher-dimensional, higher-order, forward-mode AD in the extremely general and elegant setting of calculus on manifolds and derives that implementation from a simple and precise specification. In order to motivate and discover the implementation, the paper poses the question "What does AD mean, independently of implementation?" An answer arises in the form of naturality of sampling a function and its derivative. Automatic differentiation flows out of this naturality condition, together with the chain rule. Graduating from first-order to higher-order AD corresponds to sampling all derivatives instead of just one. Next, the setting is expanded to arbitrary vector spaces, in which derivative values are linear maps. The specification of AD adapts to this elegant and very general setting, which even simplifies the development.

[1]  Simon L. Peyton Jones,et al.  Imperative functional programming , 1993, POPL '93.

[2]  Simon Peyton Jones,et al.  Playing by the rules: rewriting as a practical optimisation technique in GHC , 2001 .

[3]  Simon L. Peyton Jones,et al.  Associated type synonyms , 2005, ICFP '05.

[4]  Jerzy Karczmarczuk.  Functional Coding of Differential Forms , 1999 .

[5]  M. Douglas McIlroy,et al.  The music of streams , 2001, Inf. Process. Lett..

[6]  Graham Hutton,et al.  The generic approximation lemma , 2001, Inf. Process. Lett..

[7]  Conor McBride,et al.  Applicative programming with effects , 2008, J. Funct. Program..

[8]  Jerzy Karczmarczuk,et al.  Functional Differentiation of Computer Programs , 1998, ICFP '98.

[9]  Henrik Nilsson.  Functional automatic differentiation with dirac impulses , 2003, ICFP '03.

[10]  Barak A. Pearlmutter,et al.  Lazy multivariate higher-order forward-mode AD , 2007, POPL '07.

[11]  Conal Elliott.  Denotational design with type class morphisms , 2011 .

[12]  Barak A. Pearlmutter,et al.  Nesting forward-mode AD in a functional framework , 2008, High. Order Symb. Comput..

[13]  Barak A. Pearlmutter,et al.  Perturbation Confusion and Referential Transparency:Correct Functional Implementation of Forward-Mode AD , 2005 .

[14]  Ralf Hinze,et al.  Generalizing generalized tries , 2000, Journal of Functional Programming.

[15]  R. E. Wengert,et al.  A simple automatic derivative evaluation program , 1964, Commun. ACM.

[16]  Conal Elliott,et al.  Push-pull functional reactive programming , 2009, Haskell.