Universal Approximation Property of Neural Ordinary Differential Equations

Neural ordinary differential equations (NODEs) is an invertible neural network architecture promising for its free-form Jacobian and the availability of a tractable Jacobian determinant estimator. Recently, the representation power of NODEs has been partly uncovered: they form an $L^p$-universal approximator for continuous maps under certain conditions. However, the $L^p$-universality may fail to guarantee an approximation for the entire input domain as it may still hold even if the approximator largely differs from the target function on a small region of the input space. To further uncover the potential of NODEs, we show their stronger approximation property, namely the $\sup$-universality for approximating a large class of diffeomorphisms. It is shown by leveraging a structure theorem of the diffeomorphism group, and the result complements the existing literature by establishing a fairly large set of mappings that NODEs can approximate with a stronger guarantee.

[1]  Zuowei Shen,et al.  Deep Learning via Dynamical Systems: An Approximation Perspective , 2019, Journal of the European Mathematical Society.

[2]  Han Zhang,et al.  Approximation Capabilities of Neural ODEs and Invertible Residual Networks , 2020, ICML.

[3]  N. G. Parke,et al.  Ordinary Differential Equations. , 1958 .

[4]  T. H. Gronwall.  Note on the Derivatives with Respect to a Parameter of the Solutions of a System of Differential Equations , 1919 .

[5]  Masashi Sugiyama,et al.  Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators , 2020, NeurIPS.

[6]  Eric Nalisnick,et al.  Normalizing Flows for Probabilistic Modeling and Inference , 2019, J. Mach. Learn. Res..

[7]  John N. Mather.  Commutators of diffeomorphisms , 1974 .

[8]  Cem Anil,et al.  Sorting out Lipschitz function approximation , 2018, ICML.

[9]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[10]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[11]  J. Mather,et al.  Commutators of diffeomorphisms: II , 1975 .

[12]  D. Epstein,et al.  The simplicity of certain groups of homeomorphisms , 1970 .

[13]  William P. Thurston,et al.  Foliations and groups of diffeomorphisms , 1974 .

[14]  Yee Whye Teh,et al.  Augmented Neural ODEs , 2019, NeurIPS.

[15]  Adam M. Oberman,et al.  How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization , 2020, ICML.

[16]  W. Derrick,et al.  A Global Existence and Uniqueness Theorem for Ordinary Differential Equations , 1976, Canadian Mathematical Bulletin.

[17]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.