Deep learning as optimal control problems

Abstract We briefly review recent work where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We report here new preliminary experiments with implicit symplectic Runge-Kutta methods. In this paper, we discuss ongoing and future research in this area.

[1]  Kurt Keutzer,et al.  ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs , 2019, IJCAI.

[2]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[3]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Eldad Haber,et al.  Deep Neural Networks Motivated by Partial Differential Equations , 2018, Journal of Mathematical Imaging and Vision.

[5]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[6]  Long Chen,et al.  Maximum Principle Based Algorithms for Deep Learning , 2017, J. Mach. Learn. Res..

[7]  Carola-Bibiane Schönlieb,et al.  Deep learning as optimal control problems: models and numerical methods , 2019, Journal of Computational Dynamics.

[8]  Kurt Keutzer,et al.  ANODEV2: A Coupled Neural ODE Evolution Framework , 2019, ArXiv.

[9]  H. C. Andersen.  Rattle: A “velocity” version of the shake algorithm for molecular dynamics calculations , 1983 .

[10]  William W. Hager,et al.  Runge-Kutta methods in optimal control and the transformed adjoint system , 2000, Numerische Mathematik.

[11]  Yann LeCun,et al.  A theoretical framework for back-propagation , 1988 .

[12]  E Weinan,et al.  A Proposal on Machine Learning via Dynamical Systems , 2017, Communications in Mathematics and Statistics.

[13]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[14]  Jesús María Sanz-Serna,et al.  Symplectic Runge-Kutta Schemes for Adjoint Equations, Automatic Differentiation, Optimal Control, and More , 2015, SIAM Rev..