Calibration results for Bayesian model specication

When the goal is inference about an unknown and prediction of future data D on the basis of data D and background assumptions/judgments B, the process of Bayesian model specication involves two ingredients: the condi- tional probability distributions p( jB) and p(Dj ;B). Here we focus on specifying p(Dj ;B), and we argue that calibration considerations | paying attention to how often You get the right answer | should be an integral part of this speci- cation process. After contrasting Bayes-factor-based and predictive model-choice criteria, we present some calibration results, in xed- and random-eects Poisson models, relevant to addressing two of the basic questions that arise in Bayesian model specication: (Q1) Is model Mj better than Mj0? and (Q2) Is model Mj good enough? In particular, we show that LSF S, a full-sample log score predictive model-choice criterion, has better small-sample model discrimination performance than either DIC or a cross-validation-style log-scoring criterion, in the simulation setting we consider; we examine the large-sample behavior of LSF S; and we (a) demonstrate that the popular posterior predictive tail-area method for answering a question related to Q2 can be poorly calibrated and (b) document the success of a method for calibrating it.

[1]  B. M. Hill,et al.  Theory of Probability , 1990 .

[2]  J. Richard,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[3]  H. Akaike.  A new look at the statistical model identification , 1974 .

[4]  M. Kendall,et al.  Kendall's advanced theory of statistics , 1995 .

[5]  A. F. Smith.  Present Position and Potential Developments: Some Personal Views Bayesian Statistics , 1984 .

[6]  L. Pettit.  The Conditional Predictive Ordinate for the Normal Distribution , 1990 .

[7]  A. Dawid.  Calibration-Based Empirical Probability , 1985 .

[8]  R. Little.  Calibrated Bayes , 2006 .

[9]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[10]  A. P. Dawid,et al.  Present position and potential developments: some personal views , 1984 .

[11]  G. C. Tiao,et al.  Bayesian inference in statistical analysis , 1973 .

[12]  R. Baierlein.  Probability Theory: The Logic of Science , 2004 .

[13]  M. Aitkin.  Posterior Bayes Factors , 1991 .

[14]  Purushottam W. Laud,et al.  Predictive Model Selection , 1995 .

[15]  R. T. Cox.  The Algebra of Probable Inference , 1962 .

[16]  A. Gelfand,et al.  Bayesian Model Choice: Asymptotics and Exact Calculations , 1994 .

[17]  R. Mises,et al.  Wahrscheinlichkeit, Statistik und Wahrheit. , 1936 .

[18]  B. De Finetti,et al.  Funzione caratteristica di un fenomeno aleatorio , 1929 .

[19]  A. Dawid.  Fisherian Inference in Likelihood and Prequential Frames of Reference , 1991 .

[20]  Adrian E. Raftery,et al.  Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors , 1999 .

[21]  Alan E. Gelfand,et al.  Model choice: A minimum posterior predictive loss approach , 1998, AISTATS.

[22]  John Venn,et al.  The Logic Of Chance , 1888 .

[23]  Milovan Krnjajic,et al.  Parametric and nonparametric Bayesian model specification: A case study involving models for count data , 2008, Comput. Stat. Data Anal..

[24]  J. Berger,et al.  The Intrinsic Bayes Factor for Model Selection and Prediction , 1996 .

[25]  C. Mallows,et al.  Exchangeability and data analysis , 1993 .

[26]  Alan E. Gelfand,et al.  Model Determination using sampling-based methods , 1996 .

[27]  A. Kolmogoroff.  Grundbegriffe der Wahrscheinlichkeitsrechnung , 1933 .

[28]  R. T. Cox.  Probability, frequency and reasonable expectation , 1990 .

[29]  M. J. Bayarri,et al.  P Values for Composite Null Models , 2000 .

[30]  Lurdes Y. T. Inoue,et al.  Decision Theory: Principles and Approaches , 2009 .

[31]  L. Stein,et al.  Probability and the Weighing of Evidence , 1950 .

[32]  C. Hendriksen,et al.  Consequences of assessment and intervention among elderly people: a three year randomised controlled trial. , 1984, British medical journal.

[33]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[34]  J. Ghosh,et al.  Some Bayesian predictive approaches to model selection , 2005 .

[35]  Edward E. Leamer,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[36]  Cinzia Carota,et al.  Some Faults of the Bayes Factor in Nonparametric Model Selection , 2006, Stat. Methods Appl..

[37]  James M. Robins,et al.  Asymptotic Distribution of P Values in Composite Null Models , 2000 .

[38]  David Draper,et al.  Coherence and calibration: comments on subjectivity and "objectivity'' in Bayesian analysis (comment on articles by Berger and by Goldstein) , 2006 .

[39]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[40]  Dimitris Fouskakis,et al.  Comparing Stochastic Optimization Methods for Variable Selection in Binary Outcome Prediction, With Application to Health Policy , 2008 .

[41]  Milovan Krnjajic,et al.  Contributions to bayesian statistical analysis: model specification and nonparametric inference , 2005 .

[42]  Xiao-Li Meng,et al.  POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES , 1996 .

[43]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[44]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[46]  R. T. Cox,et al.  The Algebra of Probable Inference , 1962 .

[47]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[48]  Luis R. Pericchi,et al.  Model Selection and Hypothesis Testing based on Objective Probabilities and Bayes Factors , 2005 .