On the use of marginal posteriors in marginal likelihood estimation via importance sampling

The efficiency of a marginal likelihood estimator where the product of the marginal posterior distributions is used as an importance sampling function is investigated. The approach is generally applicable to multi-block parameter vector settings, does not require additional Markov Chain Monte Carlo (MCMC) sampling and is not dependent on the type of MCMC scheme used to sample from the posterior. The proposed approach is applied to normal regression models, finite normal mixtures and longitudinal Poisson models, and leads to accurate marginal likelihood estimates.

[1]  Tom Heskes,et al.  Discussion of ``Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations'' by H. Rue, S. Martino and N. Chopin , 2009 .

[2]  Martin D. Weinberg,et al.  Computing the Bayes Factor from a Markov chain Monte Carlo Simulation of the Posterior Distribution , 2009, 0911.1777.

[3]  Ioannis Ntzoufras,et al.  Explaining the behavior of joint and marginal Monte Carlo estimators in latent variable models with independence assumptions , 2013, Stat. Comput..

[4]  J. Simonoff.  Multivariate Density Estimation , 1996 .

[5]  M. Newton.  Approximate Bayesian-inference With the Weighted Likelihood Bootstrap , 1994 .

[6]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[7]  M. P. Hobson,et al.  Bayesian evidence for two companions orbiting HIP 5158 , 2011, 1105.1150.

[8]  A. Raftery,et al.  Estimating Bayes Factors via Posterior Simulation with the Laplace—Metropolis Estimator , 1997 .

[9]  David J. Spiegelhalter,et al.  WinBUGS user manual version 1.4 , 2003 .

[10]  Bruno Tuffin,et al.  Markov chain importance sampling with applications to rare event probability estimation , 2011, Stat. Comput..

[11]  C. Robert,et al.  Estimation of Finite Mixture Distributions Through Bayesian Sampling , 1994 .

[12]  Dimitris Karlis,et al.  Bayesian Assessment of the Distribution of Insurance Claim Counts Using Reversible Jump MCMC , 2005 .

[13]  Ming-Hui Chen,et al.  Computing marginal likelihoods from a single MCMC output , 2005 .

[14]  Xiao-Li Meng,et al.  SIMULATING RATIOS OF NORMALIZING CONSTANTS VIA A SIMPLE IDENTITY: A THEORETICAL EXPLORATION , 1996 .

[15]  Charles J. Geyer,et al.  Practical Markov Chain Monte Carlo , 1992 .

[16]  Petros Dellaportas,et al.  On Bayesian model and variable selection using MCMC , 2002, Stat. Comput..

[17]  John Geweke,et al.  Interpretation and inference in mixture models: Simple MCMC works , 2007, Comput. Stat. Data Anal..

[18]  Christian Robert,et al.  Approximating the marginal likelihood in mixture models , 2007 .

[19]  P. Diggle.  Analysis of Longitudinal Data , 1995 .

[20]  Adrian E. Raftery,et al.  Computing Normalizing Constants for Finite Mixture Models via Incremental Mixture Importance Sampling (IMIS) , 2006 .

[21]  P. Moral,et al.  Sequential Monte Carlo samplers , 2002, cond-mat/0212648.

[22]  W. Wong,et al.  The calculation of posterior distributions by data augmentation , 1987 .

[23]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[24]  Bettina Gruen.  Bayesian Mixture Models with JAGS , 2015 .

[25]  S. Chib.  Marginal Likelihood from the Gibbs Output , 1995 .

[26]  J. Skilling.  Nested sampling for general Bayesian computation , 2006 .

[27]  S. Chib,et al.  Posterior Simulation and Bayes Factors in Panel Count Data Models , 1998 .

[28]  C. Robert,et al.  Reparameterisation issues in mixture modelling and their bearing on MCMC algorithms , 1999 .

[29]  Nial Friel,et al.  Estimating the evidence – a review , 2011, 1111.1957.

[30]  J. Q. Smith,et al.  1. Bayesian Statistics 4 , 1993 .

[31]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[32]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[33]  Joshua C. C. Chan,et al.  Marginal Likelihood Estimation with the Cross-Entropy Method , 2012 .

[34]  Bernhard Schölkopf,et al.  Bayesian Model Scoring in Markov Random Fields , 2007 .

[35]  A. Rukhin.  Bayes and Empirical Bayes Methods for Data Analysis , 1997 .

[36]  J. Brian Gray,et al.  Introduction to Linear Regression Analysis , 2002, Technometrics.

[37]  S. Frühwirth-Schnatter.  Markov chain Monte Carlo Estimation of Classical and Dynamic Switching and Mixture Models , 2001 .

[38]  Radford M. Neal.  Annealed importance sampling , 1998, Stat. Comput..

[39]  M. Newton,et al.  Estimating the Integrated Likelihood via Posterior Simulation Using the Harmonic Mean Identity , 2006 .

[40]  N. G. Best,et al.  WinBUGS User Manual: Version 1.4 , 2001 .

[41]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[42]  Ming-Hui Chen.  Importance-Weighted Marginal Bayesian Posterior Density Estimation , 1994 .

[43]  A. Pettitt,et al.  Marginal likelihood estimation via power posteriors , 2008 .

[44]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[45]  Adrian E. Raftery,et al.  Bayes factors and model uncertainty , 1995 .

[46]  M. Steel,et al.  Benchmark Priors for Bayesian Model Averaging , 2001 .

[47]  Iven Van Mechelen,et al.  A Bayesian approach to the selection and testing of mixture models , 2003 .

[48]  S. Chib,et al.  Marginal Likelihood From the Metropolis–Hastings Output , 2001 .

[49]  M. Postman,et al.  Probes of large-scale structure in the Corona Borealis region. , 1986 .

[50]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[51]  C. Robert,et al.  Computational and Inferential Difficulties with Mixture Posterior Distributions , 2000 .

[52]  Man-Suk Oh.  Estimation of posterior density functions from a posterior sample , 1999 .

[53]  Walter R. Gilks,et al.  Strategies for improving MCMC , 1995 .

[54]  Sylvia Richardson,et al.  Markov Chain Monte Carlo in Practice , 1997 .

[55]  S. Frühwirth-Schnatter.  Estimating Marginal Likelihoods for Mixture and Markov Switching Models Using Bridge Sampling Techniques , 2004 .

[56]  Lennart F. Hoogerheide,et al.  A comparative study of Monte Carlo methods for efficient evaluation of marginal likelihood , 2012, Comput. Stat. Data Anal..

[57]  B. Carlin,et al.  Bayesian Model Choice Via Markov Chain Monte Carlo Methods , 1995 .

[58]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[59]  H. Rue,et al.  Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations , 2009 .

[60]  F. Feroz,et al.  MultiNest: an efficient and robust Bayesian inference tool for cosmology and particle physics , 2008, 0809.3437.

[61]  Jean-Michel Marin,et al.  Bayesian Modelling and Inference on Mixtures of Distributions , 2005 .

[62]  Merlise A. Clyde,et al.  Rao–Blackwellization for Bayesian Variable Selection and Model Averaging in Linear and Binary Regression: A Novel Data Augmentation Approach , 2011 .