Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities

In this work, we discuss practical methods for the assessment, comparison, and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate because it describes the uncertainty in the estimate. The distributions of the expected utility estimates can also be used to compare models, for example, by computing the probability of one model having a better expected utility than some other model. We propose an approach using cross-validation predictive densities to obtain expected utility estimates and Bayesian bootstrap to obtain samples from their distributions. We also discuss the probabilistic assumptions made and properties of two practical cross-validation methods, importance sampling and k-fold cross-validation. As illustrative examples, we use multilayer perceptron neural networks and gaussian processes with Markov chain Monte Carlo sampling in one toy problem and two challenging real-world problems.

[1]  M. Aitkin.  Posterior Bayes Factors , 1991 .

[2]  Edward Greenberg,et al.  A Predictive Approach to Model Selection and Multicollinearity , 1993 .

[3]  P. Burman.  A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods , 1989 .

[4]  M. Stephens.  Bayesian analysis of mixture models with an unknown number of components- an alternative to reversible jump methods , 2000 .

[5]  Jouko Lampinen,et al.  Bayesian approach for neural networks--review and case studies , 2001, Neural Networks.

[6]  Jun S. Liu,et al.  Blind Deconvolution via Sequential Imputations , 1995 .

[7]  Jun S. Liu,et al.  Monte Carlo strategies in scientific computing , 2001 .

[8]  J. Lampinen,et al.  Bayesian Input Variable Selection Using Cross-Validation Predictive Densities and Reversible Jump MCMC , 2001 .

[9]  Seymour Geisser,et al.  The Predictive Sample Reuse Method with Applications , 1975 .

[10]  B. M. Hill.  Lindley's Paradox: Comment , 1982 .

[11]  A. O'Hagan,et al.  Fractional Bayes factors for model comparison , 1995 .

[12]  David H. Wolpert,et al.  The Existence of A Priori Distinctions Between Learning Algorithms , 1996, Neural Computation.

[13]  Alan E. Gelfand,et al.  Model Determination using sampling-based methods , 1996 .

[14]  Bradley P. Carlin,et al.  Markov Chain Monte Carlo Methods for Computing Bayes Factors , 2001 .

[15]  Jouko Lampinen,et al.  On Bayesian Model Asessment and Choice Using Cross-Validation Predictive Densities , 2001 .

[16]  Hanna Järvenpää,et al.  Quality characteristics of fine aggregates and controlling their effects on concrete , 2001 .

[17]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[18]  Ioannis Ntzoufras.  Aspects of Bayesian model and variable selection using MCMC , 1999 .

[19]  Sylvia Richardson,et al.  Inference and monitoring convergence , 1995 .

[20]  S. Duane,et al.  Hybrid Monte Carlo , 1987 .

[21]  David J. C. Mackay,et al.  Introduction to Monte Carlo Methods , 1998, Learning in Graphical Models.

[22]  C. Weng,et al.  On a Second-Order Asymptotic Property of the Bayesian Bootstrap Mean , 1989 .

[23]  B. Carlin,et al.  Bayesian Model Choice Via Markov Chain Monte Carlo Methods , 1995 .

[24]  S. Sundararajan,et al.  Predictive Approaches for Choosing Hyperparameters in Gaussian Processes , 1999, Neural Computation.

[25]  Petros Dellaportas,et al.  On Bayesian model and variable selection using MCMC , 2002, Stat. Comput..

[26]  J. Geweke,et al.  Bayesian Inference in Econometric Models Using Monte Carlo Integration , 1989 .

[27]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[28]  San Cristóbal Mateo,et al.  The Lack of A Priori Distinctions Between Learning Algorithms , 1996 .

[29]  Jouko Lampinen,et al.  On Bayesian Model Assessment and Choice Using Cross-Validation Predictive Densities: Appendix , 2001 .

[30]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[31]  M. Newton,et al.  A Rank Statistics Approach to the Consistency of a General Bootstrap , 1992 .

[32]  Hong Chang,et al.  Model Determination Using Predictive Distributions with Implementation via Sampling-Based Methods , 1992 .

[33]  Jun S. Liu,et al.  Sequential Imputations and Bayesian Missing Data Problems , 1994 .

[34]  A. Gelfand,et al.  Bayesian Model Choice: Asymptotics and Exact Calculations , 1994 .

[35]  Radford M. Neal,et al.  Improving Markov chain Monte Carlo Estimators by Coupling to an Approximating Chain , 2001 .

[36]  D. Mackay,et al.  A Practical Bayesian Framework for Backprop Networks , 1991 .

[37]  Geoffrey E. Hinton,et al.  The delve manual , 1996 .

[38]  David R. Jones,et al.  Bayesian methods in health technology assessment: a review. , 2000, Health technology assessment.

[39]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[40]  Aki Vehtari,et al.  Bayesian model assessment and selection using expected utilities , 2001 .

[41]  Walter R. Gilks,et al.  Model checking and model improvement , 1995 .

[42]  Mario Peruggia,et al.  On the variability of case-deletion importance sampling weights in the Bayesian linear model , 1997 .

[43]  Xiao-Li Meng,et al.  POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES , 1996 .

[44]  J. Bernardo.  Expected Information as Expected Utility , 1979 .

[45]  Mario Peruggia,et al.  Importance Link Function Estimation for Markov Chain Monte Carlo Methods , 2000 .

[46]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[47]  Ping Zhang.  Nonparametric Importance Sampling , 1996 .

[48]  H. Künsch.  The Jackknife and the Bootstrap for General Stationary Observations , 1989 .

[49]  J. Berger,et al.  The Intrinsic Bayes Factor for Model Selection and Prediction , 1996 .

[50]  Xiao-Li Meng,et al.  Posterior Predictive Assessment of Model Fitnessvia Realized , 1995 .

[51]  M. Newton.  Approximate Bayesian-inference With the Weighted Likelihood Bootstrap , 1994 .

[52]  Efficient Nonparametric Importance Sampling for Bayesian Inference , 2002 .

[53]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Albert Y. Lo,et al.  A large sample study of the Bayesian bootstrap , 1987 .

[55]  M. Stone,et al.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[56]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[57]  J. C. Lemm.  Prior Information and Generalized Questions , 1996 .

[58]  C. Chatfield.  Model uncertainty, data mining and statistical inference , 1995 .

[59]  John Mark,et al.  Introduction to radial basis function networks , 1996 .

[60]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[61]  W. Michael Conklin,et al.  Monte Carlo Methods in Bayesian Computation , 2001, Technometrics.

[62]  Radford M. Neal.  Assessing Relevance determination methods using DELVE , 1998 .

[63]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[64]  J. C. Lemm.  Bayesian field theory , 2003 .

[65]  Edmond Chow,et al.  A cross-validatory method for dependent data , 1994 .

[66]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[67]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[68]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[69]  Donald Geman,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1984 .

[70]  L. Wasserman,et al.  Computing Bayes Factors by Combining Simulation and Asymptotic Approximations , 1997 .

[71]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[72]  Using Bayesian neural networks to classify forest scenes , 1998, Other Conferences.

[73]  S. Geisser,et al.  A Predictive Approach to Model Selection , 1979 .

[74]  Raul Cano.  On The Bayesian Bootstrap , 1992 .