Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference

The importance-weighted autoencoder (IWAE) approach of Burda et al. defines a sequence of increasingly tighter bounds on the marginal likelihood of latent variable models. Recently, Cremer et al. reinterpreted the IWAE bounds as ordinary variational evidence lower bounds (ELBO) applied to increasingly accurate variational distributions. In this work, we provide yet another perspective on the IWAE bounds. We interpret each IWAE bound as a biased estimator of the true marginal likelihood where for the bound defined on $K$ samples we show the bias to be of order O(1/K). In our theoretical analysis of the IWAE objective we derive asymptotic bias and variance expressions. Based on this analysis we develop jackknife variational inference (JVI), a family of bias-reduced estimators reducing the bias to $O(K^{-(m+1)})$ for any given m

[1]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[2]  Theofanis Karaletsos,et al.  Adversarial Message Passing For Graphical Models , 2016, ArXiv.

[3]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[4]  Max Welling,et al.  Markov Chain Monte Carlo and Variational Inference: Bridging the Gap , 2014, ICML.

[5]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[6]  Don McLeish,et al.  A general method for debiasing a Monte Carlo estimator , 2010, Monte Carlo Methods Appl..

[7]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[8]  P. Hall.  Methodology and theory for the bootstrap , 1986 .

[9]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[10]  Mark A. Girolami,et al.  Unbiased Bayes for Big Data: Paths of Partial Posteriors , 2015, ArXiv.

[11]  Shakir Mohamed,et al.  Learning in Implicit Generative Models , 2016, ArXiv.

[12]  Jakub M. Tomczak,et al.  UvA-DARE ( Digital Academic Repository ) Improving Variational Auto-Encoders using Householder Flow , 2016 .

[13]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[14]  Trevor Sharot.  The Generalized Jackknife: Finite Samples and Subsample Sizes , 1976 .

[15]  H. L. Gray,et al.  On Bias Reduction in Estimation , 1971 .

[16]  Manfred Opper,et al.  Perturbative Black Box Variational Inference , 2017, NIPS.

[17]  Dustin Tran,et al.  Deep and Hierarchical Implicit Models , 2017, ArXiv.

[18]  Qiang Liu.  Wild Variational Approximations , 2016 .

[19]  Ruslan Salakhutdinov,et al.  On the quantitative analysis of deep belief networks , 2008, ICML '08.

[20]  Peter W. Glynn,et al.  Unbiased Estimation with Square Root Convergence for SDE Models , 2015, Oper. Res..

[21]  David Duvenaud,et al.  Reinterpreting Importance-Weighted Autoencoders , 2017, ICLR.

[22]  Sebastian Nowozin,et al.  Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks , 2017, ICML.

[23]  Thomas Hofmann,et al.  A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation , 2007 .

[24]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[25]  Scott W. Linderman,et al.  Variational Sequential Monte Carlo , 2017, AISTATS.

[26]  Frank D. Wood,et al.  On the Opportunities and Pitfalls of Nesting Monte Carlo Estimators , 2017 .

[27]  Rupert G. Miller.  The jackknife-a review , 1974 .

[28]  Kenta Oono,et al.  Chainer : a Next-Generation Open Source Framework for Deep Learning , 2015 .

[29]  M. H. Quenouille.  NOTES ON BIAS IN ESTIMATION , 1956 .

[30]  M. H. Quenouille.  Approximate Tests of Correlation in Time‐Series , 1949 .

[31]  Christopher G. Small,et al.  Expansions and Asymptotics for Statistics , 2010 .

[32]  Lingyun Zhang,et al.  Sample Mean and Sample Variance , 2007 .

[33]  Yee Whye Teh,et al.  Tighter Variational Bounds are Not Necessarily Better , 2018, ICML.

[34]  Ferenc Huszár,et al.  Variational Inference using Implicit Distributions , 2017, ArXiv.