Generalization of the Ignorance Score: Continuous Ranked Version and Its Decomposition

AbstractThe Brier score (BS) and its generalizations to the multicategory ranked probability score (RPS) and to the continuous ranked probability score (CRPS) are the prominent verification measures for probabilistic forecasts. Particularly, their decompositions into measures quantifying the reliability, resolution, and uncertainty of the forecasts are attractive. Information theory sets up the natural framework for forecast verification. Recently, it has been shown that the BS is a second-order approximation of the information-based ignorance score (IGN), which also contains easily interpretable components and can also be generalized to a ranked version (RIGN). Here, the IGN, its generalizations, and decompositions are systematically discussed in analogy to the variants of the BS. Additionally, a continuous ranked IGN (CRIGN) is introduced in analogy to the CRPS. The applicability and usefulness of the conceptually appealing CRIGN are illustrated, together with an algorithm to evaluate its components rel...

[1]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .

[2]  Liam Paninski,et al.  Estimation of Entropy and Mutual Information , 2003, Neural Computation.

[3]  M. Ehrendorfer.  Vorhersage der Unsicherheit numerischer Wetterprognosen: eine Übersicht , 1997 .

[4]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[5]  B. Ahrens,et al.  On upscaling of rain-gauge data for evaluating numerical weather forecasts , 2008 .

[6]  Francisco J. Doblas-Reyes,et al.  A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes , 2005 .

[7]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[8]  Nick van de Giesen,et al.  Kullback–Leibler Divergence as a Forecast Skill Score with Classic Reliability–Resolution–Uncertainty Decomposition , 2010 .

[9]  Leonard A. Smith,et al.  Evaluating Probabilistic Forecasts Using Information Theory , 2002 .

[10]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[11]  T. Palmer.  Predicting uncertainty in forecasts of weather and climate , 2000 .

[12]  S. Weijs,et al.  Accounting for Observational Uncertainty in Forecast Verification: An Information-Theoretical View on Forecasts, Observations, and Truth , 2011 .

[13]  Timothy DelSole,et al.  Predictability and Information Theory. Part I: Measures of Predictability , 2004 .

[14]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[15]  G. North,et al.  Information Theory and Climate Prediction , 1990 .

[16]  Leonard A. Smith,et al.  Scoring Probabilistic Forecasts: The Importance of Being Proper , 2007 .

[17]  H. Hersbach.  Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems , 2000 .

[18]  S. Weijs,et al.  Why hydrological predictions should be evaluated using information theory , 2010 .

[19]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[20]  A. H. Murphy,et al.  What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting , 1993 .

[21]  Riccardo Peirolo,et al.  Information gain as a score for probabilistic forecasts , 2011 .

[22]  S. Jaun,et al.  On evaluation of ensemble precipitation forecasts with observation-based ensembles , 2007 .

[23]  A. H. Murphy.  A Note on the Ranked Probability Score , 1971 .

[24]  Riccardo Benedetti,et al.  Scoring Rules for Forecast Verification , 2010 .

[25]  Bodo Ahrens,et al.  On the Weighting of Multimodel Ensembles in Seasonal and Short-Range Weather Forecasting , 2009 .

[26]  R. Kleeman.  Measuring Dynamical Prediction Utility Using Relative Entropy , 2002 .

[27]  Harald Daan.  Sensitivity of Verification Scores to the Classification of the Predictand , 1985 .

[28]  O. Talagrand,et al.  Impact of observational error on the validation of ensemble prediction systems , 2008 .

[29]  Ian T. Jolliffe.  The impenetrable hedge: a note on propriety, equitability and consistency , 2008 .

[30]  A. H. Murphy.  A New Vector Partition of the Probability Score , 1973 .

[31]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[32]  A. H. Murphy.  On the “Ranked Probability Score” , 1969 .

[33]  O. Talagrand,et al.  Evaluation of probabilistic prediction systems for a scalar variable , 2005 .

[34]  Mark A. Liniger,et al.  The discrete brier and ranked probability skill scores , 2007 .

[35]  Simon J. Mason,et al.  On Using ``Climatology'' as a Reference Strategy in the Brier and Ranked Probability Skill Scores , 2004 .

[36]  Simon J. Mason,et al.  Understanding forecast verification statistics , 2008 .

[37]  Bodo Ahrens,et al.  Information-Based Skill Scores for Probabilistic Forecasts , 2008 .