Generalization of partitioned Runge-Kutta methods for adjoint systems

This study computes the gradient of a function of numerical solutions of ordinary differential equations (ODEs) with respect to the initial condition. The adjoint method computes the gradient approximately by solving the corresponding adjoint system numerically. In this context, Sanz-Serna [SIAM Rev., 58 (2016), pp. 3--33] showed that when the initial value problem is solved by a Runge--Kutta (RK) method, the gradient can be exactly computed by applying an appropriate RK method to the adjoint system. Focusing on the case where the initial value problem is solved by a partitioned RK (PRK) method, this paper presents a numerical method, which can be seen as a generalization of PRK methods, for the adjoint system that gives the exact gradient.

[1]  William Carlisle Thacker,et al.  The role of the Hessian matrix in fitting models to measurements , 1989 .

[2]  S. Ito,et al.  Adjoint-based exact Hessian-vector multiplication using symplectic Runge-Kutta methods , 2019, ArXiv.

[3]  E. Hairer,et al.  Geometric Numerical Integration: Structure Preserving Algorithms for Ordinary Differential Equations , 2004 .

[4]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[5]  Shin-ichi Ito,et al.  Adjoint-based exact Hessian computation , 2019 .

[6]  J. M. Sanz-Serna,et al.  Partitioned Runge-Kutta methods for separable Hamiltonian problems , 1993 .

[7]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[8]  Yuto Miyatake,et al.  Estimation of ordinary differential equation models with discretization error quantification , 2019, SIAM/ASA J. Uncertain. Quantification.

[9]  F. L. Dimet,et al.  Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects , 1986 .

[10]  Zhi Wang,et al.  The Adjoint Newton Algorithm for Large-Scale Unconstrained Optimization in Meteorology Applications , 1998, Comput. Optim. Appl..

[11]  P. Rentrop,et al.  Multirate Partitioned Runge-Kutta Methods , 2001 .

[12]  Jesús María Sanz-Serna,et al.  Symplectic Runge-Kutta Schemes for Adjoint Equations, Automatic Differentiation, Optimal Control, and More , 2015, SIAM Rev..

[13]  Colin B. Macdonald,et al.  Spatially Partitioned Embedded Runge-Kutta Methods , 2013, SIAM J. Numer. Anal..

[14]  Niles A. Pierce,et al.  An Introduction to the Adjoint Approach to Design , 2000 .

[15]  Zhi Wang,et al.  The second order adjoint analysis: Theory and applications , 1992 .

[16]  Andreas Fichtner,et al.  The adjoint method in seismology: I. Theory , 2006 .

[17]  Akinori Yamanaka,et al.  Data assimilation for massive autonomous systems based on a second-order adjoint method. , 2016, Physical review. E.

[18]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.