Summary Verification Measures and Their Interpretation for Ensemble Forecasts

AbstractEnsemble prediction systems produce forecasts that represent the probability distribution of a continuous forecast variable. Most often, the verification problem is simplified by transforming the ensemble forecast into probability forecasts for discrete events, where the events are defined by one or more threshold values. Then, skill is evaluated using the mean-square error (MSE; i.e., Brier) skill score for binary events, or the ranked probability skill score (RPSS) for multicategory events. A framework is introduced that generalizes this approach, by describing the forecast quality of ensemble forecasts as a continuous function of the threshold value. Viewing ensemble forecast quality this way leads to the interpretation of the RPSS and the continuous ranked probability skill score (CRPSS) as measures of the weighted-average skill over the threshold values. It also motivates additional measures, derived to summarize other features of a continuous forecast quality function, which can be interpret...

[1]  A. H. Murphy,et al.  Diagnostic verification of probability forecasts , 1992 .

[2]  David S. Richardson,et al.  On the effect of ensemble size on the discrete and continuous ranked probability scores , 2008 .

[3]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .

[4]  Thomas M. Hamill,et al.  Ensemble Calibration of 500-hPa Geopotential Height and 850-hPa and 2-m Temperatures Using Reforecasts , 2007 .

[5]  Guillem Candille,et al.  The Multiensemble Approach: The NAEFS Example , 2009 .

[6]  Ian T. Jolliffe,et al.  Calibration of Probabilistic Forecasts of Binary Events , 2009 .

[7]  Soroosh Sorooshian,et al.  Verification of National Weather Service Ensemble Streamflow Predictions for Water Supply Forecasting in the Colorado River Basin , 2003 .

[8]  T. Hamill,et al.  Evaluation of Eta-RSM Ensemble Probabilistic Precipitation Forecasts , 1998 .

[9]  Roberto Buizza,et al.  Impact of Ensemble Size on Ensemble Prediction , 1998 .

[10]  E. Ebert.  Ability of a Poor Man's Ensemble to Predict the Probability and Distribution of Precipitation , 2001 .

[11]  E. Grimit,et al.  Initial Results of a Mesoscale Short-Range Ensemble Forecasting System over the Pacific Northwest , 2002 .

[12]  Tempei Hashino,et al.  Distributions-oriented verification of probability forecasts for small data samples , 2003 .

[13]  Tempei Hashino,et al.  Distributions-Oriented Verification of Ensemble Streamflow Predictions , 2004 .

[14]  P. L. Houtekamer,et al.  Verification of an Ensemble Prediction System against Observations , 2007 .

[15]  A. H. Murphy.  A Note on the Ranked Probability Score , 1971 .

[16]  R. Stull,et al.  Evaluation of Probabilistic Medium-Range Temperature Forecasts from the North American Ensemble Forecast System. , 2009 .

[17]  H. Hersbach.  Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems , 2000 .

[18]  Tempei Hashino,et al.  Evaluation of bias-correction methods for ensemble streamflow volume forecasts , 2006 .

[19]  Renate Hagedorn,et al.  Probabilistic Forecast Calibration Using ECMWF and GFS Ensemble Reforecasts. Part I: Two-Meter Temperatures , 2008 .

[20]  M. Clark,et al.  Climate Index Weighting Schemes for NWS ESP-Based Seasonal Volume Forecasts , 2004 .

[21]  Arun Kumar,et al.  Simulations of the ENSO Hydroclimate Signals in the Pacific Northwest Columbia River Basin , 1999 .

[22]  F. Atger,et al.  Spatial and Interannual Variability of the Reliability of Ensemble-Based Probabilistic Forecasts: Consequences for Calibration , 2003 .

[23]  Daniel S. Wilks,et al.  Diagnostic Verification of the Climate Prediction Center Long-Lead Outlooks, 1995-98 , 2000 .

[24]  Arun Kumar,et al.  Long‐range experimental hydrologic forecasting for the eastern United States , 2002 .

[25]  K. Georgakakos,et al.  Assessment of Folsom lake response to historical and potential future climate scenarios: 1. Forecasting , 2001 .

[26]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[27]  Roberto Buizza,et al.  Quantitative Precipitation Forecasts over the United States by the ECMWF Ensemble Prediction System , 2001 .

[28]  Anton Kruger,et al.  AHPSVER: A web-based system for hydrologic forecast verification , 2007, Computational Geosciences.

[29]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[30]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[31]  Arun Kumar,et al.  Seasonal Predictions, Probabilistic Verifications, and Ensemble Size , 2001 .

[32]  F. Atger,et al.  The Skill of Ensemble Prediction Systems , 1999 .

[33]  B. Kirtman.  The COLA Anomaly Coupled Model: Ensemble ENSO Prediction , 2003 .

[34]  Michael D. Kane,et al.  Nonparametric Framework for Long‐range Streamflow Forecasting , 1992 .

[35]  K. Droegemeier,et al.  Objective Verification of the SAMEX ’98 Ensemble Forecasts , 2001 .

[36]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[37]  F. Anthony Eckel,et al.  Calibrated Probabilistic Quantitative Precipitation Forecasts Based on the MRF Ensemble , 2012 .

[38]  K. Georgakakos,et al.  Evaluation of the National Weather Service Operational Hydrologic Model and Forecasts for the American River Basin , 2006 .

[39]  G. Socher.  Conclusions References , 2000 .

[40]  Thomas M. Hamill,et al.  Verification of Eta–RSM Short-Range Ensemble Forecasts , 1997 .

[41]  Thomas M. Hamill,et al.  Comparison of Ensemble-MOS Methods Using GFS Reforecasts , 2007 .

[42]  O. Talagrand,et al.  Evaluation of probabilistic prediction systems for a scalar variable , 2005 .

[43]  Edward S. Epstein,et al.  A Scoring System for Probability Forecasts of Ranked Categories , 1969 .

[44]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .