Theoretical guidelines for a partially informed forecast examiner

The paper explores probability theory foundations behind evaluation of probabilistic forecasts. The emphasis is on a situation when the forecast examiner possesses only partially the information which was available and was used to produce a forecast. We argue that in such a situation forecasts should be judged by their conditional auto-calibration. Necessary and sufficient conditions of auto-calibration are discussed and expressed in the form of testable moment conditions. The paper also analyzes relationships between forecast calibration and forecast efficiency.

[1]  M. Hashem Pesaran,et al.  Decision‐Based Methods for Forecast Evaluation , 2007 .

[2]  Anthony S. Tay,et al.  Evaluating Density Forecasts of Inflation: The Survey of Professional Forecasters , 1997 .

[3]  Norman R. Swanson,et al.  A TEST FOR COMPARING MULTIPLE MISSPECIFIED CONDITIONAL INTERVAL MODELS , 2005, Econometric Theory.

[4]  Paul H. Kupiec,et al.  Techniques for Verifying the Accuracy of Risk Measurement Models , 1995 .

[5]  D. Hendry,et al.  Econometric Evaluation of Linear Macro-Economic Models , 1986 .

[6]  Anthony S. Tay,et al.  Multivariate Density Forecast Evaluation and Calibration In Financial Risk Management: High-Frequency Returns on Foreign Exchange , 1999, Review of Economics and Statistics.

[7]  Alexander Tsyplakov,et al.  Evaluating Density Forecasts: A Comment , 2011 .

[8]  M. Degroot.  Uncertainty, Information, and Sequential Experiments , 1962 .

[9]  Malte Knüppel,et al.  Evaluating the Calibration of Multi-Step-Ahead Density Forecasts Using Raw Moments , 2015, SSRN Electronic Journal.

[10]  Clive W. J. Granger,et al.  A Decision_Theoretic Approach to Forecast Evaluation , 2000 .

[11]  F. Sanders.  On Subjective Probability Forecasting , 1963 .

[12]  Jeremy Berkowitz.  Testing Density Forecasts, With Applications to Risk Management , 2001 .

[13]  T. Gneiting,et al.  Combining Predictive Distributions , 2011, 1106.1638.

[14]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[15]  S. Hall,et al.  Combining density forecasts , 2007 .

[16]  David R. Cox,et al.  Prediction and asymptotics , 1996 .

[17]  Kenneth F. Wallis,et al.  Scoring rules and survey density forecasts , 2011 .

[18]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[19]  D. Cox.  Tests of Separate Families of Hypotheses , 1961 .

[20]  Gianni Amisano,et al.  Comparing Density Forecasts via Weighted Likelihood Ratio Tests , 2007 .

[21]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[22]  A further analysis of the conference board’s new Leading Economic Index , 2015 .

[23]  Robert J. Shiller,et al.  Rational Expectations and the Dynamic Structure of Macroeconomic Models:A Critical Review , 1978 .

[24]  T. Bollerslev,et al.  A CONDITIONALLY HETEROSKEDASTIC TIME SERIES MODEL FOR SPECULATIVE PRICES AND RATES OF RETURN , 1987 .

[25]  H. White,et al.  A Reality Check for Data Snooping , 2000 .

[26]  David I. Harvey.  The evaluation of economic forecasts , 1997 .

[27]  Gerald S. Rogers,et al.  Mathematical Statistics: A Decision Theoretic Approach , 1967 .

[28]  Francis X. Diebold,et al.  Scoring the Leading Indicators , 2020, Business Cycles.

[29]  Ken West.  Asymptotic Inference about Predictive Ability, An Additional Appendix , 1994 .

[30]  Clive W. J. Granger,et al.  Outline of forecast theory using generalized cost functions , 1999 .

[31]  Michael P. Clements,et al.  Evaluating interval forecasts of high-frequency financial data , 2003 .

[32]  Norman R. Swanson,et al.  Predictive density and conditional confidence interval accuracy tests , 2006 .

[33]  Charles F. Manski,et al.  Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters , 2006 .

[34]  Thomas F. Cooley,et al.  ASYMPTOTIC LIKELIHOOD BASED PREDICTION FUNCTIONS , 1990 .

[35]  Leonard A. Smith,et al.  Scoring Probabilistic Forecasts: The Importance of Being Proper , 2007 .

[36]  A. H. Murphy,et al.  A General Framework for Forecast Verification , 1987 .

[37]  Halbert White,et al.  Tests of Conditional Predictive Ability , 2003 .

[38]  Jochen Bröcker,et al.  Reliability, sufficiency, and the decomposition of proper scores , 2009 .

[39]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[40]  Kenneth F. Wallis,et al.  Chi-Squared Tests of Interval and Density Forecasts, and the Bank of England's Fan Charts , 2001, SSRN Electronic Journal.

[41]  Jose A. Lopez,et al.  Methods for Evaluating Value-at-Risk Estimates , 1998 .

[42]  Giorgio Valente,et al.  Comparing the accuracy of density forecasts from competing models , 2002 .

[43]  O. Kallenberg.  Foundations of Modern Probability , 2021, Probability Theory and Stochastic Modelling.

[44]  Yi-Ting Chen.  Moment tests for density forecast evaluation in the presence of parameter estimation uncertainty , 2011 .

[45]  Michael P. Clements,et al.  Forecast Encompassing Tests and Probability Forecasts , 2010 .

[46]  Norman R. Swanson,et al.  Predictive Density Evaluation , 2005 .

[47]  Peter F. Christoffersen.  Evaluating Interval Forecasts , 1998 .

[48]  Stephen E. Fienberg,et al.  The Comparison and Evaluation of Forecasters. , 1983 .

[49]  Simone Manganelli,et al.  CAViaR , 2004 .

[50]  Michael P. Clements.  Evaluating the survey of professional forecasters probability distributions of expected inflation based on derived event probability forecasts , 2006 .

[51]  Dick van Dijk,et al.  Likelihood-based scoring rules for comparing density forecasts in tails , 2011 .

[52]  James Mitchell,et al.  Evaluating density forecasts: forecast combinations, model mixtures, calibration and sharpness , 2011 .

[53]  Anthony S. Tay,et al.  Evaluating Density Forecasts with Applications to Financial Risk Management , 1998 .

[54]  J. Geweke,et al.  Optimal Prediction Pools , 2008 .

[55]  A. Brockwell,et al.  Universal Residuals: A Multivariate Transformation. , 2007, Statistics & probability letters.

[56]  L. Hansen.  Large Sample Properties of Generalized Method of Moments Estimators , 1982 .

[57]  Anthony S. Tay,et al.  Evaluating Density Forecasts , 1997 .

[58]  Kernel-based calibration diagnostics for recession and inflation probability forecasts , 2011 .

[59]  Yong Bao,et al.  Comparing Density Forecast Models , 2007 .

[60]  Hajo Holzmann,et al.  The role of the information set for forecasting—with applications to risk management , 2014, 1404.7653.

[61]  Norman R. Swanson,et al.  Bootstrap Conditional Distribution Tests in the Presence of Dynamic Misspecification , 2003 .