Evaluating probabilistic forecasts with the R package scoringRules

Probabilistic forecasts in the form of probability distributions over future events have become popular in several fields including meteorology, hydrology, economics, and demography. In typical applications, many alternative statistical models and data sources can be used to produce probabilistic forecasts. Hence, evaluating and selecting among competing methods is an important task. The scoringRules package for R provides functionality for comparative evaluation of probabilistic models based on proper scoring rules, covering a wide range of situations in applied work. This paper discusses implementation and usage details, presents case studies from meteorology and economics, and points to the relevant background literature.

[1]  L. Held,et al.  Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds , 2008 .

[2]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[3]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[4]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[5]  Alexander Jordan,et al.  Facets of forecast evaluation , 2016 .

[6]  Gregor Kastner,et al.  Dealing with Stochastic Volatility in Time Series Using the R Package stochvol , 2016, 1906.12134.

[7]  James D. Hamilton.  A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle , 1989 .

[8]  Adrian E. Raftery,et al.  Use and communication of probabilistic forecasts , 2014, Stat. Anal. Data Min..

[9]  Lennart F. Hoogerheide,et al.  Bayesian Estimation of the GARCH(1,1) Model with Student-t Innovations , 2009, R J..

[10]  Tim N. Palmer,et al.  Ensemble forecasting , 2008, J. Comput. Phys..

[11]  Achim Zeileis,et al.  Heteroscedastic Censored and Truncated Regression with crch , 2016, R J..

[12]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[13]  T. Gneiting,et al.  The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification , 2006 .

[14]  Tilmann Gneiting,et al.  Probabilistic Forecasting and Comparative Model Assessment Based on Markov Chain Monte Carlo Output , 2016, 1608.06802.

[15]  Achim Zeileis,et al.  Extending Extended Logistic Regression: Extended versus Separate versus Ordered versus Censored , 2014 .

[16]  Adrian E. Raftery,et al.  Weather Forecasting with Ensemble Methods , 2005, Science.

[17]  A. H. Murphy,et al.  THE RANKED PROBABILITY SCORE AND THE PROBABILITY SCORE: A COMPARISON , 1970 .

[18]  T. Hamill,et al.  Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities* , 2015 .

[19]  Adrian E. Raftery,et al.  Probabilistic Weather Forecasting in R , 2011 .

[20]  M. Scheuerer.  Probabilistic quantitative precipitation forecasting using Ensemble Model Output Statistics , 2013, 1302.0893.

[21]  Sylvia Frühwirth-Schnatter,et al.  Finite Mixture and Markov Switching Models , 2006 .

[22]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[23]  Patrick Gerland,et al.  Bayesian Population Projections for the United Nations. , 2014, Statistical science : a review journal of the Institute of Mathematical Statistics.

[24]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .