Generalization of the Discrete Brier and Ranked Probability Skill Scores for Weighted Multimodel Ensemble Forecasts

Abstract This note describes how the widely used Brier and ranked probability skill scores (BSS and RPSS, respectively) can be correctly applied to quantify the potential skill of probabilistic multimodel ensemble forecasts. It builds upon the study of Weigel et al. where a revised RPSS, the so-called discrete ranked probability skill score (RPSSD), was derived, circumventing the known negative bias of the RPSS for small ensemble sizes. Since the BSS is a special case of the RPSS, a debiased discrete Brier skill score (BSSD) could be formulated in the same way. Here, the approach of Weigel et al., which so far was only applicable to single model ensembles, is generalized to weighted multimodel ensemble forecasts. By introducing an “effective ensemble size” characterizing the multimodel, the new generalized RPSSD can be expressed such that its structure becomes equivalent to the single model case. This is of practical importance for multimodel assessment studies, where the consequences of varying effective...

[1]  T. N. Krishnamurti,et al.  Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble. , 1999, Science.

[2]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[3]  Balaji Rajagopalan,et al.  Categorical Climate Forecasts through Regularization and Optimal Combination of Multiple GCM Ensembles , 2002 .

[4]  Martin Leutbecher,et al.  The Impact of Moist Singular Vectors and Horizontal Resolution on Short-Range Limited-Area Ensemble Forecasts for Two European Winter Storms , 2006 .

[5]  Simon J. Mason,et al.  On Using ``Climatology'' as a Reference Strategy in the Brier and Ranked Probability Skill Scores , 2004 .

[6]  Upmanu Lall,et al.  Improved Combination of Multiple Atmospheric GCM Ensembles for Seasonal Prediction , 2004 .

[7]  A. H. Murphy.  On the “Ranked Probability Score” , 1969 .

[8]  Mark A. Liniger,et al.  The discrete brier and ranked probability skill scores , 2007 .

[9]  Andrew P. Morse,et al.  DEVELOPMENT OF A EUROPEAN MULTIMODEL ENSEMBLE SYSTEM FOR SEASONAL-TO-INTERANNUAL PREDICTION (DEMETER) , 2004 .

[10]  Francisco J. Doblas-Reyes,et al.  Forecast assimilation: a unified framework for the combination of multi-model weather and climate predictions , 2005 .

[11]  Probabilistic high-resolution forecast of heavy precipitation over Central Europe , 2004 .

[12]  Renate Hagedorn,et al.  The rationale behind the success of multi-model ensembles in seasonal forecasting-II , 2005 .

[13]  A. Sterl,et al.  The ERA‐40 re‐analysis , 2005 .

[14]  R. T. Barnes,et al.  A performance comparison of coupled and uncoupled versions of the Met Office seasonal prediction general circulation model , 2005 .

[15]  Renate Hagedorn,et al.  The rationale behind the success of multi-model ensembles in seasonal forecasting — I. Basic concept , 2005 .

[16]  David S. Richardson,et al.  Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size , 2001 .

[17]  A. H. Murphy.  A Note on the Ranked Probability Score , 1971 .

[18]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[19]  Arun Kumar,et al.  Seasonal Predictions, Probabilistic Verifications, and Ensemble Size , 2001 .

[20]  Roberto Buizza,et al.  Impact of Ensemble Size on Ensemble Prediction , 1998 .

[21]  Francisco J. Doblas-Reyes,et al.  A Debiased Ranked Probability Skill Score to Evaluate Probabilistic Ensemble Forecasts with Small Ensemble Sizes , 2005 .

[22]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .