Conditional Exceedance Probabilities

Probabilistic forecasts of variables measured on a categorical or ordinal scale, such as precipitation occurrence or temperatures exceeding a threshold, are typically verified by comparing the relative frequency with which the target event occurs given different levels of forecast confidence. The degree to which this conditional (on the forecast probability) relative frequency of an event corresponds with the actual forecast probabilities is known as reliability, or calibration. Forecast reliability for binary variables can be measured using the Murphy decomposition of the (half) Brier score, and can be presented graphically using reliability and attributes diagrams. For forecasts of variables on continuous scales, however, an alternative measure of reliability is required. The binned probability histogram and the reliability component of the continuous ranked probability score have been proposed as appropriate verification procedures in this context, but are subject to some limitations. A procedure is proposed that is applicable in the context of forecast ensembles and is an extension of the binned probability histogram. Individual ensemble members are treated as estimates of quantiles of the forecast distribution, and the conditional probability that the observed precipitation, for example, exceeds the amount forecast [the conditional exceedance probability (CEP)] is calculated. Generalized linear regression is used to estimate these conditional probabilities. A diagram showing the CEPs for ranked ensemble members is suggested as a useful method for indicating reliability when forecasts are on a continuous scale, and various statistical tests are suggested for quantifying the reliability.

[1]  Elizabeth A. Peck,et al.  Introduction to Linear Regression Analysis , 2001 .

[2]  T. D. Mitchell,et al.  A comprehensive set of high-resolution grids of monthly climate for Europe and the globe: the observed record (1901-2000) and 16 scenarios (2001-2100). , 2004 .

[3]  David P. Rowell,et al.  Assessing Potential Seasonal Predictability with an Ensemble of Multidecadal GCM Simulations , 1998 .

[4]  A. H. Murphy,et al.  The attributes diagram A geometrical framework for assessing the quality of probability forecasts , 1986 .

[5]  Jeffrey L. Anderson.  A Method for Producing and Evaluating Probabilistic Forecasts from Ensemble Model Integrations , 1996 .

[6]  Narayanaswamy Balakrishnan,et al.  Order statistics and inference , 1991 .

[7]  M. Claussen,et al.  The atmospheric general circulation model ECHAM-4: Model description and simulation of present-day climate , 1996 .

[8]  B. Arnold,et al.  A first course in order statistics , 1994 .

[9]  B. Hunt,et al.  Chaotic influences and the problem of deterministic seasonal predictions , 1995 .

[10]  A. H. Murphy,et al.  What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting , 1993 .

[11]  Simon J. Mason,et al.  On Using ``Climatology'' as a Reference Strategy in the Brier and Ranked Probability Skill Scores , 2004 .

[12]  A. H. Murphy.  A New Vector Partition of the Probability Score , 1973 .

[13]  H. Hersbach.  Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems , 2000 .

[14]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .

[15]  A. Dawid,et al.  On Testing the Validity of Sequential Probability Forecasts , 1993 .

[16]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[17]  F. Semazzi,et al.  ENSO signals in East African rainfall seasons , 2000 .

[18]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[19]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[20]  A. H. Murphy,et al.  A Case Study of the Use of Statistical Models in Forecast Verification: Precipitation Probability Forecasts , 1998 .

[21]  Simon J. Mason,et al.  Comparison of Some Statistical Methods of Probabilistic Forecasting of ENSO. , 2002 .

[22]  Nicholas E. Graham,et al.  Conditional Probabilities, Relative Operating Characteristics, and Relative Operating Levels , 1999 .

[23]  Magne Jørgensen,et al.  When 90% confidence intervals are 50% certain: on the credibility of credible intervals , 2005 .

[24]  Kimberly L. Elmore,et al.  Alternatives to the Chi-Square Test for Evaluating Rank Histograms from Ensemble Forecasts , 2005 .

[25]  George A. F. Seber,et al.  Linear regression analysis , 1977 .

[26]  Thomas M. Hamill,et al.  Verification of Eta–RSM Short-Range Ensemble Forecasts , 1997 .

[27]  T. Hamill.  Interpretation of Rank Histograms for Verifying Ensemble Forecasts , 2001 .

[28]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[29]  F. Atger,et al.  Estimation of the reliability of ensemble‐based probabilistic forecasts , 2004 .