How much does simplification of probability forecasts reduce forecast quality?

Probability forecasts from an ensemble are often discretized into a small set of categories before being distributed to the users. This study investigates how such simplification can affect the forecast quality of probabilistic predictions as measured by the Brier score (BS). An example from the European Centre for Medium-Range Weather Forecasts (ECMWF) operational seasonal ensemble forecast system is used to show that the simplification of the forecast probabilities reduces the Brier skill score (BSS) by as much as 57% with respect to the skill score obtained with the full set of probabilities issued from the ensemble. This is more obvious for a small number of probability categories and is mainly due to a decrease in forecast resolution of up to 36%. The impact of the simplification as a function of the ensemble size is also discussed. The results suggest that forecast quality should be made available for the set of probabilities that the forecast user has access to as well as for the complete set of probabilities issued by the ensemble forecasting system. Copyright © 2008 Royal Meteorological Society

[1]  A. Sterl,et al.  The ERA‐40 re‐analysis , 2005 .

[2]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[3]  Arthur Vidard,et al.  The ECMWF System 3 ocean analysis , 2007 .

[4]  Neville Nicholls,et al.  commentary and analysis: The Insignificance of Significance Testing , 2001 .

[5]  L. Isaksen,et al.  The ERA-40 Reanalysis , 2004 .

[6]  F. Atger,et al.  Estimation of the reliability of ensemble‐based probabilistic forecasts , 2004 .

[7]  John R. Lanzante,et al.  A Cautionary Note on the Use of Error Bars , 2005 .

[8]  A. H. Murphy,et al.  A new decomposition of the Brier score: formulation and interpretation , 1986 .

[9]  Leonard A. Smith,et al.  Combining dynamical and statistical ensembles , 2003 .

[10]  David S. Richardson,et al.  Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size , 2001 .

[11]  Christopher A. T. Ferro,et al.  Comparing Probabilistic Forecasting Systems with the Brier Score , 2007 .

[12]  A. Barnston,et al.  Multimodel Ensembling in Seasonal Climate Forecasting at IRI , 2003 .

[13]  Thomas M. Hamill,et al.  Probabilistic Quantitative Precipitation Forecasts Based on Reforecast Analogs: Theory and Application , 2006 .

[14]  I. Jolliffe.  Uncertainty and Inference for Verification Measures , 2007 .

[15]  I. Jolliffe,et al.  Two Extra Components in the Brier Score Decomposition , 2008 .

[16]  Kristian Mogensen,et al.  Development of the ECMWF seasonal forecast System 3 , 2007 .

[17]  Renate Hagedorn,et al.  The rationale behind the success of multi-model ensembles in seasonal forecasting — I. Basic concept , 2005 .

[18]  Roberto Buizza,et al.  The Impact of Horizontal Resolution and Ensemble Size on Probabilistic Forecasts of Precipitation by the ECMWF Ensemble Prediction System , 2002 .

[19]  Francisco J. Doblas-Reyes,et al.  Forecast assimilation: a unified framework for the combination of multi-model weather and climate predictions , 2005 .

[20]  A. H. Murphy,et al.  The attributes diagram A geometrical framework for assessing the quality of probability forecasts , 1986 .

[21]  Renate Hagedorn,et al.  The rationale behind the success of multi-model ensembles in seasonal forecasting-II , 2005 .