Calibration tests for count data

Calibration, the statistical consistency of forecast distributions and observations, is a central requirement for probabilistic predictions. Calibration of continuous forecasts has been widely discussed, and significance tests are commonly used to detect whether a prediction model is miscalibrated. However, calibration tests for discrete forecasts are rare, especially for distributions with unlimited support. In this paper, we propose two types of calibration tests for count data: tests based on conditional exceedance probabilities and tests based on proper scoring rules. For the latter, three scoring rules are considered: the ranked probability score, the logarithmic score and the Dawid-Sebastiani score. Simulation studies show that all the different tests have good control of the type I error rate and sufficient power under miscalibration. As an illustration, we apply the methodology to weekly data on meningoccocal disease incidence in Germany, 2001–2006. The results show that the test approach is powerful in detecting miscalibrated forecasts.

[1]  S. K. Katti.  Moments of the Absolute Difference and Absolute Deviation of Discrete Distributions , 1960 .

[2]  Claudia Czado,et al.  Predictive Model Assessment for Count Data , 2009, Biometrics.

[3]  J. Heijne,et al.  Automated detection of infectious disease outbreaks: hierarchical time series models , 2006, Statistics in Medicine.

[4]  B. Leroux,et al.  Statistical models for autocorrelated count data , 2006, Statistics in medicine.

[5]  Andre Charlett,et al.  An Improved Algorithm for Outbreak Detection in Multiple Surveillance Systems , 2013, Statistics in medicine.

[6]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[7]  P. Newbold,et al.  Tests for Forecast Encompassing , 1998 .

[8]  Nick Andrews,et al.  A Statistical Algorithm for the Early Detection of Outbreaks of Infectious Disease , 1996 .

[9]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[10]  Leonhard Held,et al.  Improved auxiliary mixture sampling for hierarchical models of non-Gaussian data , 2009, Stat. Comput..

[11]  Michael Höhle,et al.  Bayesian outbreak detection algorithm for monitoring reported cases of campylobacteriosis in Germany , 2013, Biometrical journal. Biometrische Zeitschrift.

[12]  J. Elsner,et al.  Prediction Models for Annual U.S. Hurricane Counts , 2006 .

[13]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[14]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[15]  Charles Knessl,et al.  Integral representations and asymptotic expansions for Shannon and Renyi entropies , 1998 .

[16]  Jim Q. Smith,et al.  Diagnostic checks of non‐standard time series models , 1985 .

[17]  J. Hilbe.  Negative Binomial Regression: Preface , 2007 .

[18]  Norman R. Swanson,et al.  Predictive density and conditional confidence interval accuracy tests , 2006 .

[19]  L Held,et al.  A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions , 2010, Biometrics.

[20]  Anthony S. Tay,et al.  Evaluating Density Forecasts with Applications to Financial Risk Management , 1998 .

[21]  Volker Schmid,et al.  A two-component model for counts of infectious diseases. , 2005, Biostatistics.

[22]  Peter F. Christoffersen.  Evaluating Interval Forecasts , 1998 .

[23]  A. P. Dawid,et al.  Present position and potential developments: some personal views , 1984 .

[24]  J. Hilbe.  Negative Binomial Regression: Index , 2011 .

[25]  R. Winkelmann.  Econometric Analysis of Count Data , 1997 .

[26]  D. Mark,et al.  Clinical prediction models: are we building better mousetraps? , 2003, Journal of the American College of Cardiology.

[27]  L. Held,et al.  Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds , 2008 .

[28]  A. H. Murphy,et al.  Scoring rules and the evaluation of probabilities , 1996 .

[29]  D J Spiegelhalter,et al.  Probabilistic prediction in patient management and clinical trials. , 1986, Statistics in medicine.

[30]  Tilmann Gneiting,et al.  Editorial: Probabilistic forecasting , 2008 .

[31]  Leonhard Held,et al.  Modeling seasonality in space‐time infectious disease surveillance data , 2012, Biometrical journal. Biometrische Zeitschrift.

[32]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .

[33]  D. Cox.  Two further applications of a model for binary regression , 1958 .

[34]  Gael M. Martin,et al.  Bayesian predictions of low count time series , 2005 .

[35]  David Harris,et al.  Efficient probabilistic forecasts for counts , 2011 .

[36]  M. Degroot,et al.  Probability and Statistics , 2021, Examining an Operational Approach to Teaching Probability.

[37]  Paola Sebastiani,et al.  Coherent dispersion criteria for optimal experimental design , 1999 .

[38]  Leonhard Held,et al.  A statistical framework for the analysis of multivariate infectious disease surveillance counts , 2005 .

[39]  Nicholas E. Graham,et al.  Conditional Exceedance Probabilities , 2007 .

[40]  L. Held,et al.  Multivariate modelling of infectious disease surveillance data , 2008, Statistics in medicine.