On the Comparison of Interval Forecasts

We explore interval forecast comparison when the nominal confidence level is specified, but the quantiles on which intervals are based are not specified. It turns out that the problem is difficult, and perhaps unsolvable. We first consider a situation where intervals meet the Christoffersen conditions (in particular, where they are correctly calibrated), in which case the common prescription, which we rationalize and explore, is to prefer the interval of shortest length. We then allow for mis‐calibrated intervals, in which case there is a calibration‐length tradeoff. We propose two natural conditions that interval forecast loss functions should meet in such environments, and we show that a variety of popular approaches to interval forecast comparison fail them. Our negative results strengthen the case for abandoning interval forecasts in favor of density forecasts: Density forecasts not only provide richer information, but also can be readily compared using known proper scoring rules like the log predictive score, whereas interval forecasts cannot.

[1]  I. Dunsmore,et al.  Linear-loss interval estimation of location and scale parameters. , 1968, Biometrika.

[2]  R. L. Winkler.  A Decision-Theoretic Approach to Interval Estimation , 1972 .

[3]  Richard Schmalensee,et al.  An Experimental Study of Expectation Formation , 1976 .

[4]  C. Granger,et al.  Interval forecasting. An analysis based upon ARCH-quantile estimators , 1989 .

[5]  Christian P. Robert,et al.  A Paradox in Decision-Theoretic Interval Estimation , 1990 .

[6]  Christian P. Robert,et al.  The Bayesian choice , 1994 .

[7]  M. Schervish.  Theory of Statistics , 1995 .

[8]  Rob J. Hyndman,et al.  Highest‐density forecast regions for nonlinear and non‐normal time series models , 1995 .

[9]  Rob J Hyndman,et al.  Computing and Graphing Highest Density Regions , 1996 .

[10]  F. Y. Edgeworth,et al.  The theory of statistics , 1996 .

[11]  Anthony S. Tay,et al.  Evaluating Density Forecasts with Applications to Financial Risk Management , 1998 .

[12]  Peter F. Christoffersen.  Evaluating Interval Forecasts , 1998 .

[13]  Qiwei Yao,et al.  Conditional Minimum Volume Predictive Regions for Stochastic Processes , 2000 .

[14]  Stephen E. Fienberg,et al.  Testing Statistical Hypotheses , 2005 .

[15]  Evaluation and Combination of Conditional Quantile Forecasts , 2005 .

[16]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[17]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[18]  Thomas Lumley,et al.  Trading Bias for Precision: Decision Theory for Intervals and Sets , 2008 .

[19]  T. Gneiting.  Making and Evaluating Point Forecasts , 2009, 0912.0902.

[20]  T. Gneiting.  Quantiles as optimal point forecasts , 2011 .

[21]  Jason Wu,et al.  Semiparametric forecast intervals , 2012 .

[22]  Saeid Nahavandi,et al.  Closure to the Discussion of “Prediction Intervals for Short-Term Wind Farm Generation Forecasts” and “Combined Nonparametric Prediction Intervals for Wind Power Generation” and the Discussion of “Combined Nonparametric Prediction Intervals for Wind Power Generation” , 2014 .

[23]  Frank Schorfheide,et al.  Bayesian Estimation of DSGE Models , 2015 .

[24]  K. Schlag,et al.  A method to elicit beliefs as most likely intervals , 2015, Judgment and Decision Making.

[25]  J. Geweke,et al.  Prediction Using Several Macroeconomic Models , 2013, Review of Economics and Statistics.