Advances in Forecast Evaluation

This paper surveys recent developments in the evaluation of point forecasts. Taking West’s (2006) survey as a starting point, we briefly cover the state of the literature as of the time of West’s writing. We then focus on recent developments, including advancements in the evaluation of forecasts at the population level (based on true, unknown model coefficients), the evaluation of forecasts in the finite sample (based on estimated model coefficients), and the evaluation of conditional versus unconditional forecasts. We present original results in a few subject areas: the optimization of power in determining the split of a sample into in-sample and out-of-sample portions; whether the accuracy of inference in evaluation of multistep forecasts can be improved with the judicious choice of HAC estimator (it can); and the extension of West’s (1996) theory results for population-level, unconditional forecast evaluation to the case of conditional forecast evaluation.

[1]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[2]  P. Hansen.  A Test for Superior Predictive Ability , 2005 .

[3]  R. Giacomini,et al.  Detecting and Predicting Forecast Breakdowns , 2006, SSRN Electronic Journal.

[4]  P. Newbold,et al.  Tests for Forecast Encompassing , 1998 .

[5]  Paul Newbold,et al.  Testing the equality of prediction mean squared errors , 1997 .

[6]  Andrew J. Patton,et al.  Forecast Rationality Tests Based on Multi-Horizon Bounds , 2011 .

[7]  Barbara Rossi,et al.  Out-of-Sample Forecast Tests Robust to the Window Size Choice , 2011 .

[8]  Glenn D. Rudebusch,et al.  Forecasting Output with the Composite Leading Index: A Real-Time Analysis , 1991 .

[9]  S. Kozicki,et al.  Shifting endpoints in the term structure of interest rates , 2001 .

[10]  Lennart F. Hoogerheide,et al.  Comment on Forecast Rationality Tests Based on Multi-Horizon Bounds , 2012 .

[11]  J. Stock,et al.  Forecasting Output and Inflation: The Role of Asset Prices , 2001 .

[12]  Barbara Rossi,et al.  Out-of-Sample Forecast Tests Robust to the Choice of Window Size , 2011 .

[13]  S. B. Aruoba,et al.  Data Revisions are Not Well-Behaved , 2004 .

[14]  Dean Croushore,et al.  Forecasting with Real-Time Macroeconomic Data , 2006 .

[15]  David E. Rapach,et al.  In-sample vs. out-of-sample tests of stock return predictability in the context of data mining , 2006 .

[16]  L. Kilian.  Small-sample Confidence Intervals for Impulse Response Functions , 1998, Review of Economics and Statistics.

[17]  Todd E. Clark,et al.  A Bayesian evaluation of alternative models of trend inflation , 2011 .

[18]  Michael P. Clements,et al.  Real-time forecasting of inflation and output growth with autoregressive models in the presence of data revisions , 2013 .

[19]  Kenneth Rogoff,et al.  Was It Real? The Exchange Rate‐Interest Differential Relation over the Modern Floating‐Rate Period , 1988 .

[20]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[21]  E. Fama,et al.  Dividend yields and expected stock returns , 1988 .

[22]  Richard D. Porter,et al.  Is the Price Level Tied to the M2 Monetary Aggregate in the Long Run , 1991 .

[23]  James Davidson,et al.  THE FUNCTIONAL CENTRAL LIMIT THEOREM AND WEAK CONVERGENCE TO STOCHASTIC INTEGRALS I , 2000, Econometric Theory.

[24]  D. Andrews.  Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation , 1991 .

[25]  Juri Marcucci,et al.  Comparing Forecast Accuracy: A Monte Carlo Investigation , 2009 .

[26]  L. Hansen.  LARGE SAMPLE PROPERTIES OF GENERALIZED METHOD OF , 1982 .

[27]  Michael P. Clements,et al.  Real-time Forecasting of Inflation and Output Growth in the Presence of Data Revisions , 2010 .

[28]  K. West,et al.  Forecast Evaluation of Small Nested Model Sets , 2008, SSRN Electronic Journal.

[29]  Dean Croushore,et al.  A real-time data set for macroeconomists , 2001 .

[30]  Norman R. Swanson,et al.  Nonparametric Bootstrap Procedures for Predictive Inference Based on Recursive Estimation Schemes , 2005 .

[31]  Eric Ghysels,et al.  Les organisations-partenaires / The Partner Organizations , 2022 .

[32]  Richard Schmalensee,et al.  Advertising and aggregate consumption: an analysis of causality , 1980 .

[33]  Todd E. Clark,et al.  The power of tests of predictive ability in the presence of structural breaks , 2005 .

[34]  Walter Distaso,et al.  Multiple Forecast Model Evaluation , 2011 .

[35]  Todd E. Clark.  Finite‐sample properties of tests for equal forecast accuracy , 1999 .

[36]  Yi-Ting Chen.  Moment tests for density forecast evaluation in the presence of parameter estimation uncertainty , 2011 .

[37]  Todd E. Clark,et al.  Using Out-of-Sample Mean Squared Prediction Errors to Test the Martingale Difference Hypothesis , 2004 .

[38]  Lutz Kilian,et al.  Exchange Rates and Monetary Fundamentals: What Do We Learn from Long-Horizon Regressions? , 1999 .

[39]  Allan Timmermann,et al.  Choice of Sample Split in Out-of-Sample Forecast Evaluation , 2012 .

[40]  Halbert White,et al.  Tests of Conditional Predictive Ability , 2003 .

[41]  D. Hendry,et al.  Econometric Evaluation of Linear Macro-Economic Models , 1986 .

[42]  Michael W. McCracken.  Asymptotics for out of sample tests of Granger causality , 2007 .

[43]  J. Davidson.  Stochastic Limit Theory , 1994 .

[44]  Todd E. Clark,et al.  Reality Checks and Comparisons of Nested Predictive Models , 2012 .

[45]  Ray C. Fair,et al.  The Informational Context of Ex Ante Forecasts , 1989 .

[46]  Jeremy Piger,et al.  The Use and Abuse of Real-Time Data in Economic Forecasting , 2003, Review of Economics and Statistics.

[47]  Evan F. Koenig,et al.  VAR Estimation and Forecasting When Data Are Subject to Revision , 2010 .

[48]  Kenneth S. Rogoff,et al.  Can Exchange Rates Forecast Commodity Prices? , 2008 .

[49]  Parameter estimation and tests of equal forecast accuracy between non-nested models , 2004 .

[50]  Frank T. Denton,et al.  Data Mining as an Industry , 1985 .

[51]  Kenneth S. Rogoff,et al.  Exchange rate models of the seventies. Do they fit out of sample , 1983 .

[52]  E. Wilson.  The Periodogram of American Business Activity , 1934 .

[53]  The real-time predictive content of money for output , 2001 .

[54]  Inference About Predictive Ability When There are Many Predictors , 2007 .

[55]  R. Shiller,et al.  The Informational Content of Ex Ante Forecasts , 1988 .

[56]  Todd E. Clark,et al.  Evaluating Direct Multistep Forecasts , 2005 .

[57]  Donald W. K. Andrews,et al.  An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator , 1992 .

[58]  L. Kilian,et al.  In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use? , 2002, SSRN Electronic Journal.

[59]  Norman R. Swanson,et al.  The Real-Time Predictive Content of Money for Output , 2000 .

[60]  H. Moon,et al.  A Predictability Test for a Small Number of Nested Models , 2013, SSRN Electronic Journal.

[61]  Todd E. Clark,et al.  Approximately Normal Tests for Equal Predictive Accuracy in Nested Models , 2005 .

[62]  Ian Witten,et al.  Data Mining , 2000 .

[63]  Bruce E. Hansen,et al.  Convergence to Stochastic Integrals for Dependent Heterogeneous Processes , 1992, Econometric Theory.

[64]  J. Stock,et al.  Evidence on Structural Instability in Macroeconomic Time Series Relations , 1994 .

[65]  Michael W. McCracken.  Robust out-of-sample inference , 2000 .

[66]  E. P. Howrey,et al.  The Use of Preliminary Data in Econometric Forecasting , 1978 .

[67]  Todd E. Clark,et al.  Nested Forecast Model Comparisons: A New Approach to Testing Equal Accuracy , 2009 .

[68]  Simon van Norden,et al.  Série Scientifique Scientific Series the Reliability of Inflation Forecasts Based on Output Gap Estimates in Real Time the Reliability of Inflation Forecasts Based on Output Gap Estimates in Real Time* , 2022 .

[69]  David F. Hendry,et al.  Unpredictability and the Foundations of Economic Forecasting , 2004 .

[70]  W. Newey,et al.  Automatic Lag Selection in Covariance Matrix Estimation , 1994 .

[71]  Todd E. Clark,et al.  Tests of Equal Forecast Accuracy and Encompassing for Nested Models , 1999 .

[72]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[73]  J. Richard,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[74]  Ken West,et al.  Chapter 3 Forecast Evaluation , 2006 .

[75]  Norman R. Swanson,et al.  Predictive Density Evaluation , 2005 .

[76]  I. Welch,et al.  A Comprehensive Look at the Empirical Performance of Equity Premium Prediction II , 2004, SSRN Electronic Journal.

[77]  Dean Croushore,et al.  A Real-Time Data Set for Macroeconomists: Does the Data Vintage Matter? , 1999, Review of Economics and Statistics.

[78]  R. Mariano,et al.  Statistical Tests for Multiple Forecast Comparison , 2012 .

[79]  Robert,et al.  THE FUNCTIONAL CENTRAL LIMIT THEOREM AND WEAK CONVERGENCE TO STOCHASTIC INTEGRALS I Weakly Dependent Processes , 2000 .

[80]  Barbara Rossi,et al.  Recursive Predictability Tests for Real-Time Data , 2005 .

[81]  Kevin D. Hoover,et al.  Data mining reconsidered: encompassing and the general-to-specific approach to specification search , 1997 .

[82]  Todd E. Clark,et al.  Tests of equal predictive ability with Real-Time data. Discussion paper , 2007 .

[83]  Todd E. Clark,et al.  Tests of Equal Forecast Accuracy for Overlapping Models , 2011 .

[84]  Todd E. Clark.  Can Out-of-Sample Forecast Comparisons Help Prevent Overfitting? , 2000 .

[85]  J. Stock,et al.  Why Has U.S. Inflation Become Harder to Forecast , 2007 .

[86]  Todd E. Clark,et al.  Testing for Unconditional Predictive Ability , 2010 .

[87]  D. H. Mellor,et al.  Real time , 1981 .

[88]  Michael W. McCracken,et al.  Inference about predictive ability , 2007 .

[89]  Dean Croushore,et al.  A Real-Time Data Set for Macroeconomists: Does the Data Vintage Matter? , 1999 .

[90]  Kenneth Rogoff,et al.  Empirical exchange rate models of the seventies , 1983 .

[91]  Testing Predictive Ability and Power Robustification , 2012 .

[92]  David E. Runkle,et al.  Are preliminary announcements of the money stock rational forecasts , 1984 .

[93]  K. West.  Another Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator , 1995 .

[94]  R. Stambaugh,et al.  Predictive Regressions , 1999 .

[95]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[96]  A. Lo,et al.  Data-Snooping Biases in Tests of Financial Asset Pricing Models , 1989 .

[97]  Jeremy Piger,et al.  The Use and Abuse of 'Real-Time' Data in Economic Forecasting , 2000 .

[98]  Norman R. Swanson,et al.  Chapter 5 Predictive Density Evaluation , 2006 .

[99]  C. Chatfield.  Model uncertainty, data mining and statistical inference , 1995 .

[100]  Ray C. Fair,et al.  Comparing Information in Forecasts from Econometric Models , 1990 .

[101]  H. White,et al.  A Reality Check for Data Snooping , 2000 .