Hierarchical Importance Weighted Autoencoders

Importance weighted variational inference (Burda et al., 2015) uses multiple i.i.d. samples to have a tighter variational lower bound. We believe a joint proposal has the potential of reducing the number of redundant samples, and introduce a hierarchical structure to induce correlation. The hope is that the proposals would coordinate to make up for the error made by one another to reduce the variance of the importance estimator. Theoretically, we analyze the condition under which convergence of the estimator variance can be connected to convergence of the lower bound. Empirically, we confirm that maximization of the lower bound does implicitly minimize variance. Further analysis shows that this is a result of negative correlation induced by the proposed hierarchical meta sampling scheme, and performance of inference also improves when the number of samples increases.

[1]  Yoshua Bengio,et al.  Reweighted Wake-Sleep , 2014, ICLR.

[2]  Adji B. Dieng,et al.  Variational Inference via χ Upper Bound Minimization , 2017 .

[3]  Ole Winther,et al.  Auxiliary Deep Generative Models , 2016, ICML.

[4]  Dmitry Vetrov,et al.  Importance Weighted Hierarchical Variational Inference , 2019, NeurIPS.

[5]  Alexandre Lacoste,et al.  Neural Autoregressive Flows , 2018, ICML.

[6]  Nando de Freitas,et al.  Inductive Principles for Restricted Boltzmann Machine Learning , 2010, AISTATS.

[7]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[8]  Yee Whye Teh,et al.  Filtering Variational Objectives , 2017, NIPS.

[9]  Amos J. Storkey,et al.  Towards a Neural Statistician , 2016, ICLR.

[10]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[11]  Mingyuan Zhou,et al.  Semi-Implicit Variational Inference , 2018, ICML.

[12]  George Tucker,et al.  Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives , 2019, ICLR.

[13]  Aaron C. Courville,et al.  Sequentialized Sampling Importance Resampling and Scalable IWAE , 2017 .

[14]  Mike Wu,et al.  Differentiable Antithetic Sampling for Variance Reduction in Stochastic Variational Inference , 2019, AISTATS.

[15]  Andriy Mnih,et al.  Variational Inference for Monte Carlo Objectives , 2016, ICML.

[16]  David Barber,et al.  An Auxiliary Variational Method , 2004, ICONIP.

[17]  David Duvenaud,et al.  Inference Suboptimality in Variational Autoencoders , 2018, ICML.

[18]  Yee Whye Teh,et al.  Tighter Variational Bounds are Not Necessarily Better , 2018, ICML.

[19]  David Duvenaud,et al.  Reinterpreting Importance-Weighted Autoencoders , 2017, ICLR.

[20]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[21]  Dmitry P. Vetrov,et al.  Doubly Semi-Implicit Variational Inference , 2018, AISTATS.

[22]  Justin Domke,et al.  Importance Weighting and Variational Inference , 2018, NeurIPS.

[23]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[24]  Ryan P. Adams,et al.  Variational Boosting: Iteratively Refining Posterior Approximations , 2016, ICML.

[25]  Hugo Larochelle,et al.  The Neural Autoregressive Distribution Estimator , 2011, AISTATS.

[26]  Thomas Müller,et al.  Neural Importance Sampling , 2018, ACM Trans. Graph..

[27]  David Duvenaud,et al.  Joint Importance Sampling for Variational Inference , 2018 .

[28]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[29]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[30]  Thomas P. Minka,et al.  Divergence measures and message passing , 2005 .

[31]  Dustin Tran,et al.  Variational Inference via \chi Upper Bound Minimization , 2016, NIPS.

[32]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[33]  Sebastian Nowozin,et al.  Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference , 2018, ICLR.

[34]  Yee Whye Teh,et al.  Revisiting Reweighted Wake-Sleep , 2018, ArXiv.

[35]  Dustin Tran,et al.  Hierarchical Variational Models , 2015, ICML.