Variational Inference with Continuously-Indexed Normalizing Flows

Continuously-indexed flows (CIFs) have recently achieved improvements over baseline normalizing flows in a variety of density estimation tasks. In this paper, we adapt CIFs to the task of variational inference (VI) through the framework of auxiliary VI, and demonstrate that the advantages of CIFs over baseline flows can also translate to the VI setting for both sampling from posteriors with complicated topology and performing maximum likelihood estimation in latent-variable models.

[1]  Dmitry Vetrov,et al.  Importance Weighted Hierarchical Variational Inference , 2019, NeurIPS.

[2]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[3]  Yaoliang Yu,et al.  Sum-of-Squares Polynomial Flow , 2019, ICML.

[4]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[5]  Arnaud Doucet,et al.  Hamiltonian Variational Auto-Encoder , 2018, NeurIPS.

[6]  Max Welling,et al.  Multiplicative Normalizing Flows for Variational Bayesian Neural Networks , 2017, ICML.

[7]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[8]  Max Welling,et al.  Markov Chain Monte Carlo and Variational Inference: Bridging the Gap , 2014, ICML.

[9]  Shakir Mohamed,et al.  Learning in Implicit Generative Models , 2016, ArXiv.

[10]  Roger B. Grosse,et al.  On the Invertibility of Invertible Neural Networks , 2019 .

[11]  Pieter Abbeel,et al.  Compression with Flows via Local Bits-Back Coding , 2019, NeurIPS.

[12]  George Tucker,et al.  Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives , 2019, ICLR.

[13]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[16]  Mingyuan Zhou,et al.  Semi-Implicit Variational Inference , 2018, ICML.

[17]  Alexandre Lacoste,et al.  Neural Autoregressive Flows , 2018, ICML.

[18]  E. Tabak,et al.  DENSITY ESTIMATION BY DUAL ASCENT OF THE LOG-LIKELIHOOD ∗ , 2010 .

[19]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[20]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[21]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[22]  Iain Murray,et al.  Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows , 2018, AISTATS.

[23]  Ferenc Huszár,et al.  Variational Inference using Implicit Distributions , 2017, ArXiv.

[24]  Takafumi Kanamori,et al.  Density Ratio Estimation in Machine Learning , 2012 .

[25]  George Tucker,et al.  Energy-Inspired Models: Learning with Sampler-Induced Distributions , 2019, NeurIPS.

[26]  Dustin Tran,et al.  Hierarchical Variational Models , 2015, ICML.

[27]  Hongseok Yang,et al.  On Nesting Monte Carlo Estimators , 2017, ICML.

[28]  Michael U. Gutmann,et al.  Sequential Bayesian Experimental Design for Implicit Models via Mutual Information , 2020, ArXiv.

[29]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[30]  David Barber,et al.  An Auxiliary Variational Method , 2004, ICONIP.

[31]  Yee Whye Teh,et al.  Tighter Variational Bounds are Not Necessarily Better , 2018, ICML.

[32]  Emiel Hoogeboom,et al.  Integer Discrete Flows and Lossless Compression , 2019, NeurIPS.

[33]  Anthony L. Caterini,et al.  Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows , 2019, ICML.

[34]  Justin Domke,et al.  Importance Weighting and Variational Inference , 2018, NeurIPS.

[35]  Dustin Tran,et al.  Hierarchical Implicit Models and Likelihood-Free Variational Inference , 2017, NIPS.

[36]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[37]  L. Duan.  Transport Monte Carlo , 2019, 1907.10448.

[38]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[39]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[40]  Iain Murray,et al.  Neural Spline Flows , 2019, NeurIPS.

[41]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.