Posterior Predictive Assessment of Model Fitnessvia Realized

This paper considers the Bayesian counterparts of the classical tests for goodness of t and their use in judging the t of a single Bayesian model to the observed data. We focus on posterior predictive assessment, in a framework that also includes conditioning on ancillary statistics. The Bayesian formulation facilitates the construction and calculation of a meaningful reference distribution not only for any (classical) statistic, but also for any parameter-dependent \statistic" or discrepancy variable. The latter allows us to propose the realized discrepancy assessment of model tness, which directly measures the true discrepancy between data and the posited model, for any aspect of the model which we want to explore. The computation required for the realized discrepancy assessment is a straightforward byproduct of the posterior simulation used for the original Bayesian analysis. We illustrate with three applied examples. The rst example, which serves mainly to motivate the work, illustrates the diiculty of classical tests in assessing the tness of a Poisson model to a positron emission tomography image that is constrained to be nonnegative. The second and third examples illustrate the details of the posterior predictive approach in two problems: estimation in a model with inequality constraints on the parameters, and estimation in a mixture model. In all three examples, standard test statistics (either a 2 or a likelihood ratio) are not pivotal: the diiculty is not just how to compute the reference distribution for the test, but that in the classical framework no such distribution exists, independent of the unknown model parameters.

[1]  H. Chernoff.  On the Distribution of the Likelihood Ratio , 1954 .

[2]  I. Guttman.  The Use of the Concept of a Future Observation in Goodness‐Of‐Fit Problems , 1967 .

[3]  Arnold Zellner,et al.  Bayesian Analysis of Regression Error Terms , 1975 .

[4]  Edward E. Leamer,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[5]  J. Berkson.  MINIMUM CHI-SQUARE, NOT MAXIMUM LIKELIHOOD! , 1980 .

[6]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[7]  D. Rubin.  Estimation in Parallel Randomized Experiments , 1981 .

[8]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[9]  Peter McCullagh,et al.  On the asymptotic distribution of pearson's statistic in linear exponential-family models , 1985 .

[10]  Peter McCullagh,et al.  The Conditional Distribution of Goodness-of-Fit Statistics for Discrete Data , 1986 .

[11]  A. F. Smith,et al.  Statistical analysis of finite mixture distributions , 1986 .

[12]  W. Wong,et al.  The calculation of posterior distributions by data augmentation , 1987 .

[13]  W. Wong,et al.  The calculation of posterior distributions by data augmentation , 1987 .

[14]  K. Chaloner,et al.  A Bayesian approach to outlier detection and residual analysis , 1988 .

[15]  James D. Broffiti.  INCREASING AND INCREASING CONVEX BAYESIAN GRADUATION , 1988 .

[16]  Kam-Wah Tsui,et al.  Generalized p-Values in Significance Testing of Hypotheses in the Presence of Nuisance Parameters , 1989 .

[17]  N. Wermuth,et al.  On Substantive Research Hypotheses, Conditional Independence Graphs and Graphical Chain Models , 1990 .

[18]  K. Chaloner.  Bayesian residual analysis in the presence of censoring , 1991 .

[19]  Ruey S. Tsay,et al.  Model Checking Via Parametric Bootstraps in Time Series Analysis , 1992 .

[20]  Bradley P. Carlin,et al.  A SIMPLE MONTE CARLO APPROACH TO BAYESIAN GRADUATION , 1992 .

[21]  Hong Chang,et al.  Model Determination Using Predictive Distributions with Implementation via Sampling-Based Methods , 1992 .

[22]  G. C. Tiao,et al.  Robustness of maximum likelihood estimates for multi-step predictions: The exponential smoothing case , 1993 .

[23]  D. Madigan,et al.  Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam's Window , 1994 .

[24]  Xiao-Li Meng,et al.  Posterior Predictive $p$-Values , 1994 .

[25]  Andrew Gelman,et al.  Bayesian Model-Building By Pure Thought: Some Principles and Examples , 1994 .

[26]  Hal S. Stern,et al.  Using Mixture Models in Temperament Research , 1995 .

[27]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[28]  D. Rubin,et al.  The analysis of repeated-measures data on schizophrenic reaction times using mixture models. , 1995, Statistics in medicine.