Predictive Model Assessment for Count Data

Summary We discuss tools for the evaluation of probabilistic forecasts and the critique of statistical models for count data. Our proposals include a nonrandomized version of the probability integral transform, marginal calibration diagrams, and proper scoring rules, such as the predictive deviance. In case studies, we critique count regression models for patent data, and assess the predictive performance of Bayesian age‐period‐cohort models for larynx cancer counts in Germany. The toolbox applies in Bayesian or classical and parametric or nonparametric settings and to any type of ordered discrete outcomes.

[1]  I. Jolliffe,et al.  Forecast verification : a practitioner's guide in atmospheric science , 2011 .

[2]  Leonhard Held,et al.  Improved auxiliary mixture sampling for hierarchical models of non-Gaussian data , 2009, Stat. Comput..

[3]  Joachim Grammig,et al.  A new marked point process model for the federal funds rate target , 2008 .

[4]  C. Czado,et al.  Spatial modelling of claim frequency and claim size in non-life insurance , 2007 .

[5]  J. M. Sloughter,et al.  Probabilistic Quantitative Precipitation Forecasting Using Bayesian Model Averaging , 2007 .

[6]  A. Brockwell,et al.  Universal Residuals: A Multivariate Transformation. , 2007, Statistics & probability letters.

[7]  C. Czado,et al.  Zero-inflated generalized Poisson models with regression effects on the mean, dispersion and zero-inflation level applied to patent outsourcing rates , 2007 .

[8]  I. Jolliffe.  Uncertainty and Inference for Verification Measures , 2007 .

[9]  Joachim Grammig,et al.  A New Marked Point Process Model for the Federal Funds Rate Target: Methodology and Forecast Evaluation , 2007 .

[10]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[11]  Leonard A. Smith,et al.  Scoring Probabilistic Forecasts: The Importance of Being Proper , 2007 .

[12]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[13]  C. Czado,et al.  Modelling count data with overdispersion and spatial effects , 2008 .

[14]  S. Frühwirth-Schnatter,et al.  Auxiliary mixture sampling for parameter-driven models of time series of counts with applications to state space modelling , 2006 .

[15]  Re: "Bayesian projections: what are the effects of excluding data from younger age groups?". , 2006, American journal of epidemiology.

[16]  J. Elsner,et al.  Prediction Models for Annual U.S. Hurricane Counts , 2006 .

[17]  B. Leroux,et al.  Statistical models for autocorrelated count data , 2006, Statistics in medicine.

[18]  Winfried Pohlmeier,et al.  Modelling financial transaction price movements: a dynamic integer count data model , 2006 .

[19]  I. Bray,et al.  Bayesian projections: what are the effects of excluding data from younger age groups? , 2005, American journal of epidemiology.

[20]  Bruce K Armstrong,et al.  Lung cancer rate predictions using generalized additive models. , 2005, Biostatistics.

[21]  O. Talagrand,et al.  Evaluation of probabilistic prediction systems for a scalar variable , 2005 .

[22]  P. Qiu.  The Statistical Evaluation of Medical Tests for Classification and Prediction , 2005 .

[23]  Gael M. Martin,et al.  Bayesian predictions of low count time series , 2005 .

[24]  Michael P. Clements.  Evaluating Econometric Forecasts of Economic and Financial Variables , 2005 .

[25]  Sylvia Früiiwirth-Schnatter,et al.  Recursive residuals and model diagnostics for normal and non-normal state space models , 1996, Environmental and Ecological Statistics.

[26]  Russell D Wolfinger,et al.  Spatial prediction of counts and rates , 2003, Statistics in medicine.

[27]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[28]  R. Waagepetersen,et al.  Bayesian Prediction of Spatial Count Data Using Generalized Linear Mixed Models , 2002, Biometrics.

[29]  Isabelle Bray,et al.  Application of Markov chain Monte Carlo methods to projecting cancer incidence and mortality , 2002 .

[30]  H. Friedl.  Econometric Analysis of Count Data , 2002 .

[31]  Jeremy Berkowitz.  Testing Density Forecasts, With Applications to Risk Management , 2001 .

[32]  L Knorr-Held,et al.  Projections of lung cancer mortality in West Germany: a case study in Bayesian prediction. , 2001, Biostatistics.

[33]  T. Hamill.  Interpretation of Rank Histograms for Verifying Ensemble Forecasts , 2001 .

[34]  M. Handcock,et al.  Relative Distribution Methods in the Social Sciences , 1999 .

[35]  Roman Krzysztofowicz,et al.  Calibration of Probabilistic Quantitative Precipitation Forecasts , 1999 .

[36]  Paola Sebastiani,et al.  Coherent dispersion criteria for optimal experimental design , 1999 .

[37]  Alan E. Gelfand,et al.  Model choice: A minimum posterior predictive loss approach , 1998, AISTATS.

[38]  Anthony S. Tay,et al.  Evaluating Density Forecasts with Applications to Financial Risk Management , 1998 .

[39]  Martin L. Puterman,et al.  Analysis of Patent Data—A Mixed-Poisson-Regression-Model Approach , 1998 .

[40]  Noel A Cressie,et al.  Spatial modeling of snow water equivalent using covariances estimated from spatial and geomorphic attributes , 1997 .

[41]  B. Ripley,et al.  Modern Applied Statistics with S-Plus. , 1996 .

[42]  Ross Ihaka,et al.  Gentleman R: R: A language for data analysis and graphics , 1996 .

[43]  A. H. Murphy,et al.  Scoring rules and the evaluation of probabilities , 1996 .

[44]  J. Besag,et al.  Bayesian Computation and Stochastic Systems , 1995 .

[45]  Kerrie Mengersen,et al.  [Bayesian Computation and Stochastic Systems]: Rejoinder , 1995 .

[46]  Assessing the Accuracy of Time Series Model Forecasts of Count Observations: Comment , 1989 .

[47]  J. Lawless,et al.  Tests for Detecting Overdispersion in Poisson Regression Models , 1989 .

[48]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[49]  Bronwyn H Hall,et al.  The R&D Master File Documentation , 1988 .

[50]  J. Lawless.  Negative binomial and mixed Poisson regression , 1987 .

[51]  C Osmond,et al.  Using age, period and cohort models to estimate future mortality rates. , 1985, International journal of epidemiology.

[52]  Jim Q. Smith,et al.  Diagnostic checks of non‐standard time series models , 1985 .

[53]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[54]  Z. Griliches,et al.  Econometric Models for Count Data with an Application to the Patents-R&D Relationship , 1984 .

[55]  A. P. Dawid,et al.  Present position and potential developments: some personal views , 1984 .

[56]  T R Holford,et al.  The estimation of age, period and cohort effects for vital rates. , 1983, Biometrics.

[57]  S. Fienberg,et al.  Identification and estimation of age-period-cohort models in the analysis of discrete archival data , 1979 .

[58]  M. Stone.  An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[59]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[60]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[61]  Irene A. Stegun,et al.  Handbook of Mathematical Functions. , 1966 .