A new equitable score suitable for verifying precipitation in numerical weather prediction

A new equitable score is developed for monitoring precipitation forecasts and for guiding forecast system development. To accommodate the difficult distribution of precipitation, the score measures the error in ‘probability space’ through use of the climatological cumulative distribution function. For sufficiently skilful forecasting systems, the new score is less sensitive to sampling uncertainty than other established scores. It is therefore called here the ‘Stable Equitable Error in Probability Space’ (SEEPS). Weather is partitioned into three categories: ‘dry’, ‘light precipitation’ and ‘heavy precipitation’. SEEPS adapts to the climate of the region in question so that it assesses the salient aspects of the local weather, encouraging ‘refinement’ and discouraging ‘hedging’. To permit continuous monitoring of a system with resolution increasing in time, forecasts are verified against point observations. With some careful choices, observation error and lack of representativeness of model grid‐box averages are found to have relatively little impact. SEEPS can identify key forecasting errors including the overprediction of drizzle, failure to predict heavy large‐scale precipitation and incorrectly locating convective cells. Area averages are calculated taking into account the observation density. A gain of ∼2 days, at lead times of 3–9 days, over the last 14 years is found in extratropical scores of forecasts made at the European Centre for Medium‐Range Weather Forecasts (ECMWF). This gain is due to system improvements, not the increased amount of data assimilated. SEEPS may also be applicable for verifying other quantities that suffer from difficult spatio‐temporal distributions. Copyright © 2010 Royal Meteorological Society

[1]  C S Peirce,et al.  The numerical measure of the success of predictions. , 1884, Science.

[2]  P. Heidke,et al.  Berechnung Des Erfolges Und Der Güte Der Windstärkevorhersagen Im Sturmwarnungsdienst , 1926 .

[3]  Irving I. Gringorten Verification to Determine and Measure Forecasting Skill. , 1967 .

[4]  P. B. Wright ASSESSMENT OF LONG‐RANGE FORECASTS , 1976 .

[5]  Allan H. Murphy Hedging and the Mode of Expression of Weather Forecasts , 1978 .

[6]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .

[7]  A. Barnston Correspondence among the correlation, RMSE, and Heidke forecast verification measures; refinement of the Heidke score , 1992 .

[8]  Joseph P. Gerrity,et al.  A note on Gandin and Murphy's equitable skill score , 1992 .

[9]  A. H. Murphy,et al.  Equitable Skill Scores for Categorical Forecasts , 1992 .

[10]  David R. Legates,et al.  The Accuracy of United States Precipitation Data , 1994 .

[11]  J. Louis,et al.  Distortion Representation of Forecast Errors , 1995 .

[12]  Ian T. Jolliffe,et al.  Revised “LEPS” Scores for Assessing Climate Model Simulations and Long-Range Forecasts , 1996 .

[13]  Jun Du,et al.  Removal of Distortion Error from an Ensemble Forecast , 2000 .

[14]  H. Storch,et al.  Statistical Analysis in Climate Research , 2000 .

[15]  D. Stephenson Use of the “Odds Ratio” for Diagnosing Forecast Skill , 2000 .

[16]  Anna Ghelli,et al.  Verification of Precipitation Forecasts over the Alpine Region Using a High-Density Observing Network , 2002 .

[17]  D. Stephenson,et al.  A new intensity‐scale approach for the verification of spatial precipitation forecasts , 2004 .

[18]  M. Ward,et al.  Prediction of seasonal rainfall in the north nordeste of Brazil using eigenvectors of sea‐surface temperature , 2007 .

[19]  Martin Göber,et al.  Could a perfect model ever satisfy a naïve forecaster? On grid box mean versus point verification , 2008 .

[20]  N. Roberts,et al.  Scale-Selective Verification of Rainfall Accumulations from High-Resolution Forecasts of Convective Events , 2008 .

[21]  Barbara G. Brown,et al.  Forecast verification: current status and future directions , 2008 .

[22]  David B. Stephenson,et al.  The extreme dependency score: a non‐vanishing measure for forecasts of rare events , 2008 .

[23]  Robin J. Hogan,et al.  Verification of cloud‐fraction forecasts , 2009 .

[24]  I. Jolliffe,et al.  Equitability Revisited: Why the ''Equitable Threat Score'' Is Not Equitable , 2010 .