On the Computational Complexity of the Chain Rule of Differential Calculus

Many modern numerical methods in computational science and engineering rely on derivatives of mathematical models for the phenomena under investigation. The computation of these derivatives often represents the bottleneck in terms of overall runtime performance. First and higher derivative tensors need to be evaluated efficiently. The chain rule of differentiation is the fundamental prerequisite for computing accurate derivatives of composite functions which perform a potentially very large number of elemental function evaluations. Data flow dependences amongst the elemental functions give rise to a combinatorial optimization problem. We formulate Chain Rule Differentiation and we prove it to be NP-complete. Pointers to research on its approximate solution are given.

[1]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[2]  Laurent Hascoët,et al.  Mixed-language automatic differentiation , 2016, Optim. Methods Softw..

[3]  Louis B. Rall,et al.  Automatic Differentiation: Techniques and Applications , 1981, Lecture Notes in Computer Science.

[4]  Editors , 2003 .

[5]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[6]  Laurent Hascoët,et al.  Source-to-source adjoint Algorithmic Differentiation of an ice sheet model written in C , 2018, Optim. Methods Softw..

[7]  Nvmerorvm Congrventia,et al.  Disquisitiones Arithmeticae , 2017 .

[8]  R. E. Wengert,et al.  A simple automatic derivative evaluation program , 1964, Commun. ACM.

[9]  Nicolas R. Gauger,et al.  Application of Automatic Differentiation to an Incompressible URANS Solver , 2012 .

[10]  Alex Pothen,et al.  What Color Is Your Jacobian? Graph Coloring for Computing Derivatives , 2005, SIAM Rev..

[11]  Jean Utke,et al.  An Integer Programming Approach to Optimal Derivative Accumulation , 2012 .

[12]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[13]  Stéphan Thomassé,et al.  On the complexity of partial derivatives , 2016, STACS.

[14]  Andreas Griewank,et al.  Piecewise linear secant approximation via algorithmic piecewise differentiation , 2017, Optim. Methods Softw..

[15]  Andreas Griewank,et al.  Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation , 1992 .

[16]  Sadashiva S. Godbole,et al.  On Efficient Computation of Matrix Chain Products , 1973, IEEE Transactions on Computers.

[17]  Andreas Griewank,et al.  Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition , 2000, Frontiers in applied mathematics.

[18]  Uwe Naumann,et al.  Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph , 2004, Math. Program..

[19]  Uwe Naumann,et al.  The Art of Differentiating Computer Programs - An Introduction to Algorithmic Differentiation , 2012, Software, environments, tools.

[20]  Uwe Naumann,et al.  Optimal Jacobian accumulation is NP-complete , 2007, Math. Program..

[21]  Bernhard Kainz,et al.  Automatic Differentiation for GPU-Accelerated 2D/3D Registration , 2008 .

[22]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.