Deep Diffeomorphic Normalizing Flows

The Normalizing Flow (NF) models a general probability density by estimating an invertible transformation applied on samples drawn from a known distribution. We introduce a new type of NF, called Deep Diffeomorphic Normalizing Flow (DDNF). A diffeomorphic flow is an invertible function where both the function and its inverse are smooth. We construct the flow using an ordinary differential equation (ODE) governed by a time-varying smooth vector field. We use a neural network to parametrize the smooth vector field and a recursive neural network (RNN) for approximating the solution of the ODE. Each cell in the RNN is a residual network implementing one Euler integration step. The architecture of our flow enables efficient likelihood evaluation, straightforward flow inversion, and results in highly flexible density estimation. An end-to-end trained DDNF achieves competitive results with state-of-the-art methods on a suite of density estimation and variational inference tasks. Finally, our method brings concepts from Riemannian geometry that, we believe, can open a new research direction for neural density estimation.

[1]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[2]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[3]  W. Gordon.  On the Diffeomorphisms of Euclidean Space , 1972 .

[4]  Leslie G. Valiant,et al.  The Complexity of Computing the Permanent , 1979, Theor. Comput. Sci..

[5]  J. Dormand,et al.  A family of embedded Runge-Kutta formulae , 1980 .

[6]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  M. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .

[8]  Barak A. Pearlmutter.  Fast Exact Multiplication by the Hessian , 1994, Neural Computation.

[9]  W. Rossmann.  Lie Groups: An Introduction through Linear Groups , 2002 .

[10]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[11]  Alain Trouvé,et al.  Computing Large Deformation Metric Mappings via Geodesic Flows of Diffeomorphisms , 2005, International Journal of Computer Vision.

[12]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[13]  Vincent Arsigny,et al.  Processing Data in Lie Groups : An Algebraic Approach. Application to Non-Linear Registration and Diffusion Tensor MRI. (Traitement de données dans les groupes de Lie : une approche algébrique. Application au recalage non-linéaire et à l'imagerie du tenseur de diffusion) , 2006 .

[14]  Nicholas Ayache,et al.  A Log-Euclidean Framework for Statistics on Diffeomorphisms , 2006, MICCAI.

[15]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[16]  Nicholas Ayache,et al.  Non-parametric Diffeomorphic Image Registration with the Demons Algorithm , 2007, MICCAI.

[17]  L. Younes.  Shapes and Diffeomorphisms , 2010 .

[18]  Hugo Larochelle,et al.  The Neural Autoregressive Distribution Estimator , 2011, AISTATS.

[19]  Tim Salimans,et al.  Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression , 2012, ArXiv.

[20]  Hugo Larochelle,et al.  RNADE: The real-valued neural autoregressive density-estimator , 2013, NIPS.

[21]  Karol Gregor,et al.  Neural Variational Inference and Learning in Belief Networks , 2014, ICML.

[22]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[23]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[24]  P. Thomas Fletcher,et al.  Bayesian principal geodesic analysis for estimating intrinsic diffeomorphic image variability , 2015, Medical Image Anal..

[25]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[26]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[27]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Hugo Larochelle,et al.  Neural Autoregressive Distribution Estimation , 2016, J. Mach. Learn. Res..

[29]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[30]  Max Welling,et al.  Improving Variational Auto-Encoders using Householder Flow , 2016, ArXiv.

[31]  Ryan P. Adams,et al.  Early Stopping as Nonparametric Variational Inference , 2015, AISTATS.

[32]  Dustin Tran,et al.  Automatic Differentiation Variational Inference , 2016, J. Mach. Learn. Res..

[33]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[34]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[35]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[36]  Asok Ray,et al.  Principles of Riemannian Geometry in Neural Networks , 2017, NIPS.

[37]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[38]  Max Welling,et al.  Sylvester Normalizing Flows for Variational Inference , 2018, UAI.

[39]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[40]  Alexandre Lacoste,et al.  Neural Autoregressive Flows , 2018, ICML.

[41]  Theofanis Karaletsos,et al.  Pathwise Derivatives for Multivariate Distributions , 2018, AISTATS.