Performance study of marginal posterior density estimation via Kullback-Leibler divergence

In this article, we introduce Kullback-Leibler (K-L) divergence as a performance measure of marginal posterior density estimation. We show that the K-L divergence can be used to compare two density estimators as well as to assess convergence of a marginal density estimator. We also examine performance of the importance-weighted marginal density estimation (IWMDE) proposed by Chen (1994) under the K-L divergence and we further extend the IWMDE to some more complex Bayesian models where the kernel method, which is widely used for estimating marginal densities using Markov chain Monte Carlo (MCMC) sampling outputs is not applicable. Finally, we use a constrained linear multiple regression model as an example to illustrate our methodology.

[1]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[2]  Charles J. Geyer.  Conditioning in Markov Chain Monte Carlo , 1995 .

[3]  J. Simonoff.  Multivariate Density Estimation , 1996 .

[4]  S. Hu.  THE STRONG UNIFORM CONSISTENCY OF KERNEL DENSITY ESTIMATES FOR φ—MIXING SAMPLE , 1993 .

[5]  L. Wasserman,et al.  Computing Bayes Factors Using a Generalization of the Savage-Dickey Density Ratio , 1995 .

[6]  Ming-Hui Chen.  Importance-Weighted Marginal Bayesian Posterior Density Estimation , 1994 .

[7]  F. Götze,et al.  The Berry-Esseen bound for student's statistic , 1996 .

[8]  Ing Rj Ser.  Approximation Theorems of Mathematical Statistics , 1980 .

[9]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[10]  L. Tierney.  Markov Chains for Exploring Posterior Distributions , 1994 .

[11]  Ming-Hui Chen,et al.  Monte Carlo methods for Bayesian analysis of constrained parameter problems , 1998 .

[12]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  P. Hall.  On Kullback-Leibler loss and density estimation , 1987 .

[14]  Ming-Hui Chen,et al.  Bayesian Analysis for a Constrained Linear Multiple Regression Problem for Predicting the New Crop of Apples , 1996 .

[15]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[16]  Peter Hall,et al.  On Global Properties of Variable Bandwidth Density Estimators , 1992 .

[17]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[18]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[19]  Adrian F. M. Smith,et al.  Bayesian Analysis of Constrained Parameter and Truncated Data Problems , 1991 .

[20]  Weighted distributions viewed in the context of model selection: A Bayesian perspective , 1996 .

[21]  Valen E. Johnson,et al.  A Technique for Estimating Marginal Posterior Densities in Hierarchical Models Using Mixtures of Conditional Densities , 1992 .

[22]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[23]  S. Chib.  Marginal Likelihood from the Gibbs Output , 1995 .