Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

AbstractProper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper the authors present an alternative class of proper scoring rules based on the geostatistical concept of variograms. The sensitivity of these variogram-based scoring rules to incorrectly pre...

[1]  N. Cressie,et al.  Robust estimation of the variogram: I , 1980 .

[2]  R. Bruno,et al.  Geostatistical Characterization of Fractal Models of Surfaces , 1989 .

[3]  Jeffrey L. Anderson.  A Method for Producing and Evaluating Probabilistic Forecasts from Ensemble Model Integrations , 1996 .

[4]  Thomas M. Hamill,et al.  Verification of Eta–RSM Short-Range Ensemble Forecasts , 1997 .

[5]  Noel A Cressie,et al.  The Variance-Based Cross-Variogram: You Can Add Apples and Oranges , 1998 .

[6]  Thomas M. Hamill,et al.  Hypothesis Tests for Evaluating Numerical Precipitation Forecasts , 1999 .

[7]  Paola Sebastiani,et al.  Coherent dispersion criteria for optimal experimental design , 1999 .

[8]  H. Hersbach.  Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems , 2000 .

[9]  T. Hamill.  Interpretation of Rank Histograms for Verifying Ensemble Forecasts , 2001 .

[10]  P. Houtekamer,et al.  A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation , 2001 .

[11]  J. Whitaker,et al.  Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter , 2001 .

[12]  Leonard A. Smith,et al.  Evaluating Probabilistic Forecasts Using Information Theory , 2002 .

[13]  J. Whitaker,et al.  Ensemble Forecasts and the Properties of Flow-Dependent Analysis-Error Covariance Singular Vectors , 2003 .

[14]  Daniel S. Wilks,et al.  The minimum spanning tree histogram as a verification tool for multidimensional ensemble forecasts , 2004 .

[15]  Leonard A. Smith,et al.  Extending the Limits of Ensemble Forecast Verification with the Minimum Spanning Tree , 2004 .

[16]  R. Buizza,et al.  A Comparison of the ECMWF, MSC, and NCEP Global Ensemble Prediction Systems , 2005 .

[17]  X. Emery.  Variograms of Order ω: A Tool to Validate a Bivariate Distribution Model , 2005 .

[18]  John M. Lewis,et al.  Roots of Ensemble Forecasting , 2005 .

[19]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[20]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[21]  Adrian E. Raftery,et al.  Combining Spatial Statistical and Ensemble Information in Probabilistic Weather Forecasts , 2007 .

[22]  Martin Leutbecher,et al.  Scale‐dependent verification of ensemble forecasts , 2008 .

[23]  Tim N. Palmer,et al.  Ensemble forecasting , 2008, J. Comput. Phys..

[24]  L. Held,et al.  Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds , 2008 .

[25]  T. Gneiting.  Making and Evaluating Point Forecasts , 2009, 0912.0902.

[26]  Tilmann Gneiting,et al.  Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression , 2010 .

[27]  Robin Girard,et al.  Evaluating the quality of scenarios of short-term wind power generation , 2012 .

[28]  T. Thorarinsdottir,et al.  Assessing the Calibration of High-Dimensional Ensemble Forecasts Using Rank Histograms , 2013, 1310.0236.

[29]  J. Whitaker,et al.  NOAA's Second-Generation Global Medium-Range Ensemble Reforecast Dataset , 2013 .

[30]  David B. Stephenson,et al.  Three recommendations for evaluating climate predictions , 2013 .

[31]  T. Gneiting,et al.  Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling , 2013, 1302.7149.

[32]  Tilmann Gneiting,et al.  Copula Calibration , 2013, 1307.7650.

[33]  Pierre Pinson,et al.  Discrimination ability of the Energy score , 2013 .

[34]  T. Thorarinsdottir,et al.  Spatial Postprocessing of Ensemble Forecasts for Temperature Using Nonhomogeneous Gaussian Regression , 2014, 1407.0058.

[35]  D. Wilks.  Multivariate ensemble Model Output Statistics using empirical copulas , 2015 .