Simulation-based comparison of multivariate ensemble post-processing methods

Many practical applications of statistical postprocessing methods for ensemble weather forecasts require accurate modeling of spatial, temporal, and inter-variable dependencies. Over the past years, a variety of approaches has been proposed to address this need. We provide a comprehensive review and comparison of state-of-the-art methods for multivariate ensemble post-processing. We focus on generally applicable two-step approaches where ensemble predictions are first post-processed separately in each margin and multivariate dependencies are restored via copula functions in a second step. The comparisons are based on simulation studies tailored to mimic challenges occurring in practical applications and allow ready interpretation of the effects of different types of misspecifications in the mean, variance, and covariance structure of the ensemble forecasts on the performance of the post-processing methods. Overall, we find that the Schaake shuffle provides a compelling benchmark that is difficult to outperform, whereas the forecast quality of parametric copula approaches and variants of ensemble copula coupling strongly depend on the misspecifications at hand.

[1]  Thordis L. Thorarinsdottir,et al.  Multivariate probabilistic forecasting using ensemble Bayesian model averaging and copulas , 2012, 1202.3956.

[2]  Saralees Nadarajah,et al.  Modeling Annual Extreme Precipitation in China Using the Generalized Extreme Value Distribution , 2007 .

[3]  Christopher A. T. Ferro,et al.  A comparison of ensemble post‐processing methods for extreme events , 2014 .

[4]  J. Smith,et al.  Stochastic modeling of flood peaks using the generalized extreme value distribution , 2002 .

[5]  Bert Van Schaeybroeck,et al.  Ensemble post‐processing using member‐by‐member approaches: theoretical aspects , 2015 .

[6]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[7]  Achim Zeileis,et al.  Remember the past: A comparison of time-adaptive training schemes for non-homogeneous regression , 2019, Nonlinear Processes in Geophysics.

[8]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[9]  Christopher A. T. Ferro,et al.  Regime‐dependent statistical post‐processing of ensemble forecasts , 2019, Quarterly Journal of the Royal Meteorological Society.

[10]  A. Jenkinson.  The frequency distribution of the annual maximum (or minimum) values of meteorological elements , 1955 .

[11]  Alexander Jordan,et al.  Evaluating Probabilistic Forecasts with scoringRules , 2017, Journal of Statistical Software.

[12]  T. Thorarinsdottir,et al.  Assessing the Calibration of High-Dimensional Ensemble Forecasts Using Rank Histograms , 2013, 1310.0236.

[13]  Adrian E. Raftery,et al.  Probabilistic quantitative precipitation field forecasting using a two-stage spatial model , 2008, 0901.3484.

[14]  J. Bröcker.  Evaluating raw ensembles with the continuous ranked probability score , 2012 .

[15]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[16]  T. Hamill,et al.  Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities* , 2015 .

[17]  J. Lygeros,et al.  Effect of Wind Correlation on Aircraft Conflict Probability , 2007 .

[18]  D. Wilks.  Multivariate ensemble Model Output Statistics using empirical copulas , 2015 .

[19]  Annette Möller,et al.  Bivariate ensemble model output statistics approach for joint forecasting of wind speed and temperature , 2015, Meteorology and Atmospheric Physics.

[20]  Dimitri Solomatine,et al.  A Stratified Sampling Approach for Improved Sampling from a Calibrated Ensemble Forecast Distribution , 2016 .

[21]  Roman Schefzik,et al.  Ensemble calibration with preserved correlations: unifying and comparing ensemble copula coupling and member‐by‐member postprocessing , 2017 .

[22]  S. Vannitsem,et al.  Statistical Postprocessing for Weather Forecasts: Review, Challenges, and Avenues in a Big Data World , 2020, Bulletin of the American Meteorological Society.

[23]  Stephan Rasp,et al.  Neural networks for post-processing ensemble weather forecasts , 2018, Monthly Weather Review.

[24]  Achim Zeileis,et al.  Bivariate Gaussian models for wind vectors in a distributional regression framework , 2019, Advances in Statistical Climatology, Meteorology and Oceanography.

[25]  Roman Schefzik,et al.  Ensemble Postprocessing Methods Incorporating Dependence Structures , 2018 .

[26]  T. Gneiting,et al.  Ensemble Model Output Statistics for Wind Vectors , 2012, 1201.2612.

[27]  Olivier Mestre,et al.  Calibrated Ensemble Forecasts Using Quantile Regression Forests and Ensemble Model Output Statistics , 2016 .

[28]  Sebastian Lerch,et al.  Preface: Advances in post-processing and blending of deterministic and ensemble forecasts , 2020 .

[29]  Florian Ziel,et al.  Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules , 2019, 1910.07325.

[30]  Bill Ravens,et al.  An Introduction to Copulas , 2000, Technometrics.

[31]  Pierre Pinson,et al.  Generation of Scenarios from Calibrated Ensemble Forecasts with a Dual-Ensemble Copula-Coupling Approach , 2015, 1511.05877.

[32]  M. Clark,et al.  The Schaake Shuffle: A Method for Reconstructing Space–Time Variability in Forecasted Precipitation and Temperature Fields , 2004 .

[33]  D. Wilks,et al.  The weather generation game: a review of stochastic weather models , 1999 .

[34]  MatrixMarne,et al.  Modeling and Generating Random Vectorswith Arbitrary Marginal Distributions and Correlation , 1997 .

[35]  T. Thorarinsdottir,et al.  Spatial Postprocessing of Ensemble Forecasts for Temperature Using Nonhomogeneous Gaussian Regression , 2014, 1407.0058.

[36]  S. Baran,et al.  Mixture EMOS model for calibrating ensemble forecasts of wind speed , 2015, Environmetrics.

[37]  T. Gneiting,et al.  Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling , 2013, 1302.7149.

[38]  Huifen Chen,et al.  Initialization for NORTA: Generation of Random Vectors with Specified Marginals and Correlations , 2001, INFORMS J. Comput..

[39]  Robin Girard,et al.  Evaluating the quality of scenarios of short-term wind power generation , 2012 .

[40]  Tilmann Gneiting,et al.  Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression , 2010 .

[41]  Daniel S. Wilks,et al.  On assessing calibration of multivariate ensemble forecasts , 2017 .

[42]  T. Thorarinsdottir,et al.  Comparison of non-homogeneous regression models for probabilistic wind speed forecasting , 2013, 1305.2026.

[43]  Thomas M. Hamill,et al.  A method for preferential selection of dates in the Schaake shuffle approach to constructing spatiotemporal forecast fields of temperature and precipitation , 2017 .

[44]  Annette Moller,et al.  Joint probabilistic forecasting of wind speed and temperature using Bayesian model averaging , 2014, 1404.3681.

[45]  M. Scheuerer.  Probabilistic quantitative precipitation forecasting using Ensemble Model Output Statistics , 2013, 1302.0893.

[46]  Roman Schefzik,et al.  A Similarity-Based Implementation of the Schaake Shuffle , 2015, 1507.02079.

[47]  Pierre Pinson,et al.  Application of Postprocessing for Renewable Energy , 2018 .

[48]  L. Held,et al.  Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds , 2008 .

[49]  M. Sklar.  Fonctions de repartition a n dimensions et leurs marges , 1959 .

[50]  Rob J Hyndman,et al.  Automatic Time Series Forecasting: The forecast Package for R , 2008 .

[51]  Stephan Hemri,et al.  Analog‐Based Postprocessing of Navigation‐Related Hydrological Ensemble Forecasts , 2017 .

[52]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .