Comparing Possibly Misspecified Forecasts

Abstract Recent work has emphasized the importance of evaluating estimates of a statistical functional (such as a conditional mean, quantile, or distribution) using a loss function that is consistent for the functional of interest, of which there is an infinite number. If forecasters all use correctly specified models free from estimation error, and if the information sets of competing forecasters are nested, then the ranking induced by a single consistent loss function is sufficient for the ranking by any consistent loss function. This article shows, via analytical results and realistic simulation-based analyses, that the presence of misspecified models, parameter estimation error, or nonnested information sets, leads generally to sensitivity to the choice of (consistent) loss function. Thus, rather than merely specifying the target functional, which narrows the set of relevant loss functions only to the class of loss functions consistent for that functional, forecast consumers or survey designers should specify the single specific loss function that will be used to evaluate forecasts. An application to survey forecasts of U.S. inflation illustrates the results.

[1]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[2]  Andrew J. Patton.  Volatility Forecast Comparison Using Imperfect Volatility Proxies , 2006 .

[3]  M. Carvalho,et al.  Discussion of "Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations and Forecast Rankings," by Ehm, W., Gneiting, T., Jordan, A., and Krüger , 2016 .

[4]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[5]  Andrew J. Patton,et al.  Properties of Optimal Forecasts under Asymmetric Loss and Nonlinearity , 2007 .

[6]  Roger Koenker,et al.  Quantile Regression : Penalized from : Handbook of Quantile Regression , 2018 .

[7]  B. Hansen.  Autoregressive Conditional Density Estimation , 1994 .

[8]  Hajo Holzmann,et al.  The role of the information set for forecasting—with applications to risk management , 2014, 1404.7653.

[9]  T. Gneiting,et al.  Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules , 2011 .

[10]  T. Bollerslev,et al.  Generalized autoregressive conditional heteroskedasticity , 1986 .

[11]  Marco Saerens,et al.  Building cost functions minimizing to some summary statistics , 2000, IEEE Trans. Neural Networks Learn. Syst..

[12]  F. Diebold,et al.  Optimal Prediction Under Asymmetric Loss , 1994, Econometric Theory.

[13]  T. Gneiting.  Quantiles as optimal point forecasts , 2011 .

[14]  Clive W. J. Granger.  Prediction with a generalized cost of error function , 2001 .

[15]  Ivana Komunjer,et al.  Quasi-maximum likelihood estimation for conditional quantiles , 2005 .

[16]  A. Timmermann,et al.  Economic Forecasting , 2007 .

[17]  Spyros Skouras,et al.  Decisionmetrics: A Decision-Based Approach to Econometric Modelling , 2007 .

[18]  Mark Steyvers,et al.  Choosing a Strictly Proper Scoring Rule , 2013, Decis. Anal..

[19]  T. Gneiting.  Making and Evaluating Point Forecasts , 2009, 0912.0902.

[20]  L. Bregman.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming , 1967 .

[21]  Stephen E. Fienberg,et al.  Studies in Bayesian econometrics and statistics : in honor of Leonard J. Savage , 1975 .

[22]  Peter Christoffersen,et al.  Série Scientifique Scientific Series the Importance of the Loss Function in Option Valuation the Importance of the Loss Function in Option Valuation , 2022 .

[23]  Xin Guo,et al.  On the optimality of conditional expectation as a Bregman predictor , 2005, IEEE Trans. Inf. Theory.

[24]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[25]  Andrew A. Weiss,et al.  Estimating Time Series Models Using the Relevant Cost Function , 1996 .

[26]  C. Gouriéroux,et al.  PSEUDO MAXIMUM LIKELIHOOD METHODS: THEORY , 1984 .

[27]  William Thomson,et al.  Eliciting production possibilities from a well-informed manager , 1979 .

[28]  A. Zellner.  Bayesian Estimation and Prediction Using Asymmetric Loss Functions , 1986 .

[29]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[30]  Robert P. Lieli,et al.  Optimal Binary Prediction for Group Decision Making , 2010 .

[31]  Robert P. Lieli,et al.  Unrestricted and controlled identification of loss functions: Possibility and impossibility results , 2019, International Journal of Forecasting.

[32]  L. Bauwens,et al.  Handbook of Volatility Models and Their Applications: Bauwens/Handbook of Volatility Models and Their Applications , 2012 .

[33]  Tilmann Gneiting,et al.  Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings , 2015, 1503.08195.

[34]  Jonathan H. Wright,et al.  Forecasting Inflation , 2011 .

[35]  K. West,et al.  A Utility Based Comparison of Some Models of Exchange Rate Volatility , 1992 .

[36]  Robert P. Lieli,et al.  ON THE RECOVERABILITY OF FORECASTERS’ PREFERENCES , 2013, Econometric Theory.

[37]  Luc BAUWENS,et al.  Handbook of Volatility Models and Their Applications , 2012 .

[38]  Charles F. Manski,et al.  Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters , 2006 .

[39]  G. Elliott,et al.  Forecasting Conditional Probabilities of Binary Outcomes under Misspecification , 2016, Review of Economics and Statistics.