Augmented Neural ODEs

We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.

[1]  E. Coddington,et al.  Theory of Ordinary Differential Equations , 1955 .

[2]  L. Younes.  Shapes and Diffeomorphisms , 2010 .

[3]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[4]  A. Ambrosetti,et al.  A Textbook on Ordinary Differential Equations , 2014 .

[5]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[8]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[9]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[10]  E Weinan,et al.  A Proposal on Machine Learning via Dynamical Systems , 2017, Communications in Mathematics and Statistics.

[11]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Bin Dong,et al.  Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations , 2017, ICML.

[13]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[14]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[15]  Stanley Osher,et al.  EnResNet: ResNet Ensemble via the Feynman-Kac Formalism , 2018, ArXiv.

[16]  Stefanie Jegelka,et al.  ResNet with one-neuron hidden layers is a Universal Approximator , 2018, NeurIPS.

[17]  Ullrich Köthe,et al.  Analyzing Inverse Problems with Invertible Neural Networks , 2018, ICLR.

[18]  David Duvenaud,et al.  Invertible Residual Networks , 2018, ICML.

[19]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[20]  Eldad Haber,et al.  Deep Neural Networks Motivated by Partial Differential Equations , 2018, Journal of Mathematical Imaging and Vision.