Interpretation of point forecasts with unknown directive

Point forecasts can be interpreted as functionals (i.e., point summaries) of predictive distributions. We consider the situation where forecasters' directives are hidden and develop methodology for the identification of the unknown functional based on time series data of point forecasts and associated realizations. Focusing on the natural cases of state-dependent quantiles and expectiles, we provide a generalized method of moments estimator for the functional, along with tests of optimality relative to information sets that are specified by instrumental variables. Using simulation, we demonstrate that our optimality test is better calibrated and more powerful than existing solutions. In empirical examples, Greenbook gross domestic product (GDP) forecasts of the US Federal Reserve and model output for precipitation from the European Centre for Medium-Range Weather Forecasts (ECMWF) are indicative of overstatement in anticipation of extreme events.

[1]  F. Diebold,et al.  Optimal Prediction Under Asymmetric Loss , 1994, Econometric Theory.

[2]  David Kjellberg.  Measuring Expectations , 1999, Identification Problems in the Social Sciences.

[3]  Asymmetry and Federal Reserve Forecasts , 2013 .

[4]  C. Capistrán.  Bias in Federal Reserve Inflation Forecasts: Is the Federal Reserve Irrational or Just Cautious? , 2005 .

[5]  W. Newey,et al.  Large sample estimation and hypothesis testing , 1986 .

[6]  Richard P. Larrick,et al.  The wisdom of select crowds. , 2014, Journal of personality and social psychology.

[7]  Hajo Holzmann,et al.  The role of the information set for forecasting—with applications to risk management , 2014, 1404.7653.

[8]  Robert P. Lieli,et al.  Unrestricted and controlled identification of loss functions: Possibility and impossibility results , 2019, International Journal of Forecasting.

[9]  Christian Pierdzioch,et al.  Oil price forecasting under asymmetric loss , 2013 .

[10]  T. Gneiting.  Making and Evaluating Point Forecasts , 2009, 0912.0902.

[11]  Efe A. Ok.  Real analysis with economic applications , 2007 .

[12]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[13]  Bruce E. Hansen,et al.  Consistent Covariance Matrix Estimation for Dependent Heterogeneous Processes , 1992 .

[14]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[15]  Francesco Ravazzolo,et al.  Forecaster's Dilemma: Extreme Events and Forecast Evaluation , 2015, 1512.09244.

[16]  L. Hansen.  Large Sample Properties of Generalized Method of Moments Estimators , 1982 .

[17]  I. Ibragimov,et al.  Some Limit Theorems for Stationary Processes , 1962 .

[18]  Tae-Hwy Lee,et al.  Asymmetric Loss in the Greenbook and the Survey of Professional Forecasters , 2014 .

[19]  Andrew Harvey,et al.  Quantiles, expectiles and splines , 2009 .

[20]  F. Diebold,et al.  Further results on forecasting and model selection under asymmetric loss , 1996 .

[21]  Christian Pierdzioch,et al.  Forecasting the Brazilian Real and the Mexican Peso: Asymmetric Loss, Forecast Rationality, and Forecaster Herding , 2012 .

[22]  H. Tong,et al.  Asymmetric least squares regression estimation: A nonparametric approach ∗ , 1996 .

[23]  Michael T. Kiley,et al.  A comparison of forecast performance between federal reserve staff forecasts, simple reduced‐form models, and a DSGE model , 2010 .

[24]  Steffen Andersen,et al.  Lost in State Space: Are Preferences Stable? , 2008 .

[25]  Andrew J. Patton,et al.  Testing Forecast Optimality Under Unknown Loss , 2007 .

[26]  J. Horowitz,et al.  Identification and estimation of statistical functionals using incomplete data , 2006 .

[27]  R. Król,et al.  Evaluating state revenue forecasting under a flexible loss function , 2013 .

[28]  Allan Timmermann,et al.  Biases in Macroeconomic Forecasts: Irrationality or Asymmetric Loss? , 2008 .

[29]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[30]  Andrew J. Patton.  Comparing Possibly Misspecified Forecasts , 2020, Journal of Business & Economic Statistics.

[31]  M. Kynn.  The ‘heuristics and biases’ bias in expert elicitation , 2007 .

[32]  Konrad Paul Körding,et al.  The loss function of sensorimotor learning. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[33]  Tilmann Gneiting,et al.  Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings , 2015, 1503.08195.

[34]  Charles F. Manski,et al.  Interpreting Point Predictions: Some Logical Issues , 2016 .

[35]  Jan Beran,et al.  Statistics for long-memory processes , 1994 .

[36]  Johanna F. Ziegel,et al.  Higher order elicitability and Osband’s principle , 2015, 1503.08123.

[37]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[38]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[39]  Robert P. Lieli,et al.  ON THE RECOVERABILITY OF FORECASTERS’ PREFERENCES , 2013, Econometric Theory.

[40]  Olivier Coibion,et al.  Information Rigidity and the Expectations Formation Process: A Simple Framework and New Facts , 2010, SSRN Electronic Journal.

[41]  Charles F. Manski,et al.  Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters , 2006 .

[42]  Kemal Guler,et al.  Mincer–Zarnowitz quantile and expectile regressions for forecast evaluations under aysmmetric loss functions , 2017 .

[43]  A. Müller,et al.  Generalized Quantiles as Risk Measures , 2013 .

[44]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[45]  Johanna F. Ziegel,et al.  Cross-calibration of probabilistic forecasts , 2015 .

[46]  Allan Timmermann,et al.  Persistence in forecasting performance and conditional combination strategies , 2006 .

[47]  F. Diebold,et al.  Forecast Evaluation and Combination , 1996 .

[48]  D. Andrews.  Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation , 1991 .

[49]  Allan Timmermann,et al.  Estimation and Testing of Forecast Rationality under Flexible Loss , 2005 .

[50]  Halbert White,et al.  Tests of Conditional Predictive Ability , 2003 .

[51]  Siyu Zhang,et al.  Elicitation and Identification of Properties , 2014, COLT.

[52]  W. Newey,et al.  Asymmetric Least Squares Estimation and Testing , 1987 .

[53]  W. Greene,et al.  计量经济分析 = Econometric analysis , 2009 .

[54]  M. Carvalho,et al.  Discussion of "Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations and Forecast Rankings," by Ehm, W., Gneiting, T., Jordan, A., and Krüger , 2016 .

[55]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[56]  Federal reserve forecasts: asymmetry and state-dependence , 2013 .

[57]  A. Timmermann,et al.  Economic Forecasting , 2007 .

[58]  Kent Osband,et al.  Providing incentives for better cost forecasting , 1985 .

[59]  David I. Harvey.  The evaluation of economic forecasts , 1997 .

[60]  Spyros Skouras,et al.  Decisionmetrics: A Decision-Based Approach to Econometric Modelling , 2007 .

[61]  Andrew J. Patton,et al.  Forecast Rationality Tests Based on Multi-Horizon Bounds , 2011 .

[62]  Michael T. Owyang,et al.  Multivariate Forecast Evaluation and Rationality Testing , 2007, Review of Economics and Statistics.

[63]  Chris R Sims,et al.  The cost of misremembering: Inferring the loss function in visual working memory. , 2015, Journal of vision.

[64]  W. Newey,et al.  Automatic Lag Selection in Covariance Matrix Estimation , 1994 .

[65]  P. Billingsley,et al.  Probability and Measure , 1980 .

[66]  P. Burridge,et al.  A Very Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix , 1991 .

[67]  K. Lahiri,et al.  Measuring Forecast Uncertainty by Disagreement: The Missing Link , 2008 .

[68]  Evaluation and Combination of Conditional Quantile Forecasts , 2005 .

[69]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[70]  E. Mamatzakis,et al.  An assessment of the EU growth forecasts under asymmetric preferences , 2008 .

[71]  J. Thepaut,et al.  The ERA‐Interim reanalysis: configuration and performance of the data assimilation system , 2011 .

[72]  C. Heinrich,et al.  The mode functional is not elicitable , 2014 .

[73]  Andrew J. Patton,et al.  Properties of Optimal Forecasts under Asymmetric Loss and Nonlinearity , 2007 .

[74]  Andrew J. Patton,et al.  Generalized Forecast Errors, A Change of Measure, and Forecast Optimality , 2008 .

[75]  T. Gneiting,et al.  Combining Predictive Distributions , 2011, 1106.1638.

[76]  J. Davidson.  Stochastic Limit Theory: An Introduction for Econometricians , 1994 .

[77]  Jonathan H. Wright,et al.  GMM WITH WEAK IDENTIFICATION , 2000 .

[78]  Peter Bauer,et al.  The quiet revolution of numerical weather prediction , 2015, Nature.

[79]  M. C. Jones.  Expectiles and M-quantiles are quantiles , 1994 .