Normalizing Flows on Riemannian Manifolds

We consider the problem of density estimation on Riemannian manifolds. Density estimation on manifolds has many applications in fluid-mechanics, optics and plasma physics and it appears often when dealing with angular variables (such as used in protein folding, robot limbs, gene-expression) and in general directional statistics. In spite of the multitude of algorithms available for density estimation in the Euclidean spaces $\mathbf{R}^n$ that scale to large n (e.g. normalizing flows, kernel methods and variational approximations), most of these methods are not immediately suitable for density estimation in more general Riemannian manifolds. We revisit techniques related to homeomorphisms from differential geometry for projecting densities to sub-manifolds and use it to generalize the idea of normalizing flows to more general Riemannian manifolds. The resulting algorithm is scalable, simple to implement and suitable for use with automatic differentiation. We demonstrate concrete examples of this method on the n-sphere $\mathbf{S}^n$.

[1]  Ben Calderhead,et al.  Riemannian Manifold Hamiltonian Monte Carlo , 2009, 0907.1100.

[2]  Inderjit S. Dhillon,et al.  Clustering on the Unit Hypersphere using von Mises-Fisher Distributions , 2005, J. Mach. Learn. Res..

[3]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[4]  Ole Winther,et al.  Auxiliary Deep Generative Models , 2016, ICML.

[5]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[6]  Adi Ben-Israel,et al.  The Change-of-Variables Formula Using Matrix Volume , 1999, SIAM J. Matrix Anal. Appl..

[7]  Daan Wierstra,et al.  One-Shot Generalization in Deep Generative Models , 2016, ICML.

[8]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[9]  Bruno Pelletier.  Kernel density estimation on Riemannian manifolds , 2005 .

[10]  Scott W. Linderman,et al.  Rejection Sampling Variational Inference , 2016 .

[11]  M. Berger,et al.  Differential Geometry: Manifolds, Curves, and Surfaces , 1987 .

[12]  Inderjit S. Dhillon,et al.  Generative model-based clustering of directional data , 2003, KDD '03.

[13]  Adi Ben-Israel.  An application of the matrix volume in probability , 2000 .

[14]  Yiming Yang,et al.  Von Mises-Fisher Clustering Models , 2014, ICML.

[15]  Geoffrey E. Hinton,et al.  Attend, Infer, Repeat: Fast Scene Understanding with Generative Models , 2016, NIPS.

[16]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[17]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[18]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[19]  Max Welling,et al.  Markov Chain Monte Carlo and Variational Inference: Bridging the Gap , 2014, ICML.

[20]  Ole Winther,et al.  Indexable Probabilistic Matrix Factorization for Maximum Inner Product Search , 2016, AAAI.

[21]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[22]  M. Girolami,et al.  Riemann manifold Langevin and Hamiltonian Monte Carlo methods , 2011, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[23]  Alex Graves,et al.  Stochastic Backpropagation through Mixture Density Distributions , 2016, ArXiv.