Learning Neural Event Functions for Ordinary Differential Equations

The existing Neural ODE formulation relies on an explicit knowledge of the termination time. We extend Neural ODEs to implicitly defined termination criteria modeled by neural event functions, which can be chained together and differentiated through. Neural Event ODEs are capable of modeling discrete (instantaneous) changes in a continuous-time system, without prior knowledge of when these changes should occur or how many such changes should exist. We test our approach in modeling hybrid discrete- and continuous- systems such as switching dynamical systems and collision in multi-body systems, and we propose simulation-based training of point processes with applications in discrete control.

[1]  Michael I. Jordan,et al.  Nonparametric Bayesian Learning of Switching Linear Dynamical Systems , 2008, NIPS.

[2]  L. F Abbott,et al.  Lapicque’s introduction of the integrate-and-fire model neuron (1907) , 1999, Brain Research Bulletin.

[3]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[4]  Travis E. Oliphant,et al.  Guide to NumPy , 2015 .

[5]  et al.,et al.  Jupyter Notebooks - a publishing format for reproducible computational workflows , 2016, ELPUB.

[6]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[7]  E A J F Peters,et al.  Rejection-free Monte Carlo sampling for general potentials. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[8]  Yee Whye Teh,et al.  Augmented Neural ODEs , 2019, NeurIPS.

[9]  Xingjian Li,et al.  OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport , 2020, ArXiv.

[10]  K. Ito,et al.  On State Estimation in Switching Environments , 1970 .

[11]  Harold R. Parks,et al.  The Implicit Function Theorem , 2002 .

[12]  Andrew Gordon Wilson,et al.  Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data , 2020, ICML.

[13]  L. S. Pontryagin,et al.  Mathematical Theory of Optimal Processes , 1962 .

[14]  Kunal Gupta,et al.  Neural Mesh Flow: 3D Manifold Mesh Generationvia Diffeomorphic Flows , 2020, NeurIPS.

[15]  Guido Rossum,et al.  Python Reference Manual , 2000 .

[16]  Edward De Brouwer,et al.  GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series , 2019, NeurIPS.

[17]  Scott W. Linderman,et al.  Recurrent switching linear dynamical systems , 2016, 1610.08466.

[18]  Tianqi Chen,et al.  Training Deep Nets with Sublinear Memory Cost , 2016, ArXiv.

[19]  Gaël Varoquaux,et al.  The NumPy Array: A Structure for Efficient Numerical Computation , 2011, Computing in Science & Engineering.

[20]  Kazuyuki Aihara,et al.  Fully Neural Network based Model for General Temporal Point Processes , 2019, NeurIPS.

[21]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[22]  Artem Molchanov,et al.  Generalized Inner Loop Meta-Learning , 2019, ArXiv.

[23]  M. L. Chambers.  The Mathematical Theory of Optimal Processes , 1965 .

[24]  Leonidas J. Guibas,et al.  CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations , 2020, NeurIPS.

[25]  Austin R. Benson,et al.  Neural Jump Stochastic Differential Equations , 2019, NeurIPS.

[26]  Han Zhang,et al.  Approximation Capabilities of Neural ODEs and Invertible Residual Networks , 2020, ICML.

[27]  Wulfram Gerstner,et al.  Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition , 2014 .

[28]  Sekhar Tatikonda,et al.  Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE , 2020, ICML.

[29]  Amit Chakraborty,et al.  Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control , 2020, ICLR.

[30]  A. Hodgkin,et al.  The components of membrane conductance in the giant axon of Loligo , 1952, The Journal of physiology.

[31]  Ming-Yu Liu,et al.  PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[33]  Carol S. Woodward,et al.  Enabling New Flexibility in the SUNDIALS Suite of Nonlinear and Differential/Algebraic Equation Solvers , 2020, ACM Trans. Math. Softw..

[34]  Alexandr Katrutsa,et al.  Interpolated Adjoint Method for Neural ODEs , 2020, ArXiv.

[35]  L. Shampine,et al.  Event location for ordinary differential equations , 2000 .

[36]  Jason Eisner,et al.  The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process , 2016, NIPS.

[37]  Jeffrey D. Scargle,et al.  An Introduction to the Theory of Point Processes, Vol. I: Elementary Theory and Methods , 2004, Technometrics.

[38]  Yann LeCun,et al.  A theoretical framework for back-propagation , 1988 .

[39]  Kurt Keutzer,et al.  ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs , 2019, IJCAI.

[40]  Michael Figurnov,et al.  Monte Carlo Gradient Estimation in Machine Learning , 2019, J. Mach. Learn. Res..

[41]  M. Athans,et al.  State Estimation for Discrete Systems with Switching Parameters , 1978, IEEE Transactions on Aerospace and Electronic Systems.

[42]  David Duvenaud,et al.  Neural Networks with Cheap Differential Operators , 2019, NeurIPS.

[43]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[44]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[45]  William T. Shaw,et al.  Quantile mechanics , 2008, European Journal of Applied Mathematics.

[46]  Ulrike Goldschmidt,et al.  An Introduction To The Theory Of Point Processes , 2016 .

[47]  Travis E. Oliphant,et al.  Python for Scientific Computing , 2007, Computing in Science & Engineering.

[48]  A. Doucet,et al.  The Bouncy Particle Sampler: A Nonreversible Rejection-Free Markov Chain Monte Carlo Method , 2015, 1510.02451.

[49]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[50]  B. Adams,et al.  Dynamic multidrug therapies for hiv: optimal and sti control approaches. , 2004, Mathematical biosciences and engineering : MBE.

[51]  Niles A. Pierce,et al.  An Introduction to the Adjoint Approach to Design , 2000 .

[52]  Jason Yosinski,et al.  Hamiltonian Neural Networks , 2019, NeurIPS.

[53]  Wes McKinney,et al.  Python for Data Analysis , 2012 .

[54]  Adrian Sandu,et al.  FATODE: A Library for Forward, Adjoint, and Tangent Linear Integration of ODEs , 2014, SIAM J. Sci. Comput..

[55]  Yann Le Cun,et al.  A Theoretical Framework for Back-Propagation , 1988 .

[56]  David Duvenaud,et al.  Latent ODEs for Irregularly-Sampled Time Series , 2019, ArXiv.

[57]  Patrick Kidger,et al.  Neural Controlled Differential Equations for Irregular Time Series , 2020, NeurIPS.

[58]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[59]  E Weinan,et al.  Monge-Ampère Flow for Generative Modeling , 2018, ArXiv.