Information-Based Skill Scores for Probabilistic Forecasts

Abstract The information content, that is, the predictive capability, of a forecast system is often quantified with skill scores. This paper introduces two ranked mutual information skill (RMIS) scores, RMISO and RMISY, for the evaluation of probabilistic forecasts. These scores are based on the concept of mutual information of random variables as developed in information theory. Like the ranked probability skill score (RPSS)—another and often applied skill score—the new scores compare cumulative probabilities for multiple event thresholds. The RMISO quantifies the fraction of information in the observational data that is explained by the forecasts. The RMISY quantifies the amount of useful information in the forecasts. Like the RPSS, the new scores are biased, but they can be debiased with a simple and robust method. This and additional promising characteristics of the scores are discussed with ensemble forecast assessment experiments.

[1]  Simon J. Mason,et al.  On Using ``Climatology'' as a Reference Strategy in the Brier and Ranked Probability Skill Scores , 2004 .

[2]  Martin Ehrendorfer,et al.  Bayesian Approach to Decision Making Using Ensemble Weather Forecasts , 2006 .

[3]  Harald Daan.  Sensitivity of Verification Scores to the Classification of the Predictand , 1985 .

[4]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[5]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[6]  R. Stuhlmann,et al.  On the validation of the atmospheric model REMO with ISCCP data and precipitation measurements using simple statistics , 1998 .

[7]  G. North,et al.  Information Theory and Climate Prediction , 1990 .

[8]  Leonard A. Smith,et al.  Evaluating Probabilistic Forecasts Using Information Theory , 2002 .

[9]  Valerie Isham,et al.  Some models for rainfall based on stochastic point processes , 1987, Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences.

[10]  T. Palmer.  Predicting uncertainty in forecasts of weather and climate , 2000 .

[11]  Tiziana Paccagnella,et al.  The COSMO-LEPS mesoscale ensemble system: validation of the methodology and verification , 2005 .

[12]  Francisco J. Doblas-Reyes,et al.  A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes , 2005 .

[13]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[14]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[15]  B. Ahrens.  Evaluation of precipitation forecasting with the limited area model ALADIN in an alpine watershed , 2003 .

[16]  Luca G. Lanza,et al.  Correlation patterns and information flows in rainfall fields , 2006 .

[17]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[18]  Mark A. Liniger,et al.  Generalization of the Discrete Brier and Ranked Probability Skill Scores for Weighted Multimodel Ensemble Forecasts , 2007 .

[19]  Liam Paninski,et al.  Estimation of Entropy and Mutual Information , 2003, Neural Computation.

[20]  Thomas Schürmann.  Bias analysis in entropy estimation , 2004 .

[21]  Mark A. Liniger,et al.  The discrete brier and ranked probability skill scores , 2007 .

[22]  M. Ehrendorfer.  Vorhersage der Unsicherheit numerischer Wetterprognosen: eine Übersicht , 1997 .

[23]  F. Atger,et al.  The Skill of Ensemble Prediction Systems , 1999 .

[24]  A. H. Murphy.  The Finley Affair: A Signal Event in the History of Forecast Verification , 1996 .

[25]  Timothy DelSole,et al.  Predictability and Information Theory. Part I: Measures of Predictability , 2004 .

[26]  R. Selten.  Axiomatic Characterization of the Quadratic Scoring Rule , 1998 .

[27]  P. S. Eagleson.  Ecohydrology: Darwinian Expression of Vegetation Form and Function , 2002 .

[28]  A. H. Murphy.  A New Vector Partition of the Probability Score , 1973 .

[29]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[30]  Ga Miller,et al.  Note on the bias of information estimates , 1955 .

[31]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .