Variational Bayes With Intractable Likelihood

ABSTRACT Variational Bayes (VB) is rapidly becoming a popular tool for Bayesian inference in statistical modeling. However, the existing VB algorithms are restricted to cases where the likelihood is tractable, which precludes their use in many interesting situations such as in state--space models and in approximate Bayesian computation (ABC), where application of VB methods was previously impossible. This article extends the scope of application of VB to cases where the likelihood is intractable, but can be estimated unbiasedly. The proposed VB method therefore makes it possible to carry out Bayesian inference in many statistical applications, including state--space models and ABC. The method is generic in the sense that it can be applied to almost all statistical models without requiring too much model-based derivation, which is a drawback of many existing VB algorithms. We also show how the proposed method can be used to obtain highly accurate VB approximations of marginal posterior distributions. Supplementary material for this article is available online.

[1]  M. Pitt,et al.  Importance Sampling Squared for Bayesian Inference in Latent Variable Models , 2013, 1309.3339.

[2]  Sean Gerrish,et al.  Black Box Variational Inference , 2013, AISTATS.

[3]  Hagai Attias,et al.  Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.

[4]  Tim Salimans,et al.  Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression , 2012, ArXiv.

[5]  M. Wand,et al.  Gaussian Variational Approximate Inference for Generalized Linear Mixed Models , 2012 .

[6]  Matthew P. Wand,et al.  Fully simplified multivariate normal updates in non-conjugate variational message passing , 2014, J. Mach. Learn. Res..

[7]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[8]  Juha Karhunen,et al.  Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes , 2010, J. Mach. Learn. Res..

[9]  M. Pitt,et al.  Likelihood analysis of non-Gaussian measurement time series , 1997 .

[10]  C. Andrieu,et al.  The pseudo-marginal approach for efficient Monte Carlo computations , 2009, 0903.5480.

[11]  R. Kohn,et al.  The Block Pseudo-Marginal Sampler , 2016, 1603.02485.

[12]  M. Beaumont.  Estimation of population growth or decline in genetically monitored populations. , 2003, Genetics.

[13]  N. Shephard,et al.  BAYESIAN INFERENCE BASED ONLY ON SIMULATED LIKELIHOOD: PARTICLE FILTER ANALYSIS OF DYNAMIC ECONOMIC MODELS , 2011, Econometric Theory.

[14]  Ralph S. Silva,et al.  On Some Properties of Markov Chain Monte Carlo Simulation Methods Based on the Particle Filter , 2012 .

[15]  N. Laird,et al.  A likelihood-based method for analysing longitudinal binary responses , 1993 .

[16]  J. McCulloch,et al.  Simple consistent estimators of stable distribution parameters , 1986 .

[17]  M. Wand,et al.  Mean field variational Bayes for continuous sparse signal shrinkage: Pitfalls and remedies , 2014 .

[18]  A. Doucet,et al.  Particle Markov chain Monte Carlo methods , 2010 .

[19]  J. L. Nolan.  Stable Distributions. Models for Heavy Tailed Data , 2001 .

[20]  Nicolas Chopin,et al.  Expectation Propagation for Likelihood-Free Inference , 2011, 1107.5959.

[21]  Richard E. Turner,et al.  Two problems with variational expectation maximisation for time-series models , 2011 .

[22]  R. Kohn,et al.  Annealed Important Sampling for Models with Latent Variables , 2014, 1402.6035.

[23]  Jin-Chuan Duan,et al.  Density-Tempered Marginalized Sequential Monte Carlo Samplers , 2013 .

[24]  Gareth W. Peters,et al.  Likelihood-free Bayesian inference for α-stable models , 2012, Comput. Stat. Data Anal..

[25]  P. Donnelly,et al.  Inferring coalescence times from DNA sequence data. , 1997, Genetics.

[26]  D. M. Titterington,et al.  Variational approximations in Bayesian model selection for finite mixture distributions , 2007, Comput. Stat. Data Anal..

[27]  J. Ware,et al.  Applied Longitudinal Analysis , 2004 .

[28]  P. Moral.  Feynman-Kac Formulae: Genealogical and Interacting Particle Systems with Applications , 2004 .

[29]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[30]  A. Owen.  Monte Carlo Variance of Scrambled Net Quadrature , 1997 .

[31]  Geoffrey E. Hinton,et al.  Variational Learning for Switching State-Space Models , 2000, Neural Computation.

[32]  Michael I. Jordan,et al.  Variational Bayesian Inference with Stochastic Search , 2012, ICML.

[33]  David Wingate,et al.  Automated Variational Inference in Probabilistic Programming , 2013, ArXiv.

[34]  R. Kohn,et al.  Regression Density Estimation With Variational Methods and Stochastic Approximation , 2012 .

[35]  Siem Jan Koopman,et al.  Time Series Analysis by State Space Methods , 2001 .

[36]  J. Sacks.  Asymptotic Distribution of Stochastic Approximation Procedures , 1958 .

[37]  J. Durbin,et al.  Monte Carlo maximum likelihood estimation for non-Gaussian state space models , 1997 .

[38]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[39]  Linda S. L. Tan,et al.  Variational Inference for Generalized Linear Mixed Models Using Partially Noncentered Parametrizations , 2012, 1205.3906.

[40]  F. Pillichshammer,et al.  Digital Nets and Sequences: Discrepancy Theory and Quasi-Monte Carlo Integration , 2010 .

[41]  H. Haario,et al.  An adaptive Metropolis algorithm , 2001 .

[42]  Robert Kohn,et al.  Block-Wise Pseudo-Marginal Metropolis-Hastings , 2016 .

[43]  A. Doucet,et al.  Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator , 2012, 1210.1871.

[44]  Jean-Michel Marin,et al.  Approximate Bayesian computational methods , 2011, Statistics and Computing.

[45]  A. Doucet,et al.  The correlated pseudomarginal method , 2015, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[46]  Andras Fulop,et al.  Density-Tempered Marginalized Sequential Monte Carlo Samplers , 2015 .

[47]  N. Shephard,et al.  Stochastic Volatility: Likelihood Inference And Comparison With Arch Models , 1996 .