FORECAST EVALUATION

Contents Abstract 100 Keywords 100 1. Introduction 101 2. A brief history 102 3. A small number of nonnested models, Part I 104 4. A small number of nonnested models, Part II 106 5. A small number of nonnested models, Part III 111 6. A small number of models, nested: MSPE 117 7. A small number of models, nested, Part II 122 8. Summary on small number of models 125 9. Large number of models 125 10. Conclusions 131 Acknowledgements 132 References 132 Abstract This chapter summarizes recent literature on asymptotic inference about forecasts. Both analytical and simulation based methods are discussed. The emphasis is on techniques applicable when the number of competing models is small. Techniques applicable when a large number of models is compared to a benchmark are also briefly discussed.

[1]  Norman R. Swanson,et al.  OUT-OF-SAMPLE TESTS FOR GRANGER CAUSALITY , 2001, Macroeconomic Dynamics.

[2]  Todd E. Clark,et al.  Evaluating Direct Multistep Forecasts , 2005 .

[3]  Sydney C. Ludvigson,et al.  Consumption, Aggregate Wealth and Expected Stock Returns , 1999 .

[4]  Ching-Kang Ing,et al.  MULTISTEP PREDICTION IN AUTOREGRESSIVE PROCESSES , 2003, Econometric Theory.

[5]  A note on in‐sample and out‐of‐sample tests for Granger causality , 2005 .

[6]  Stephen Taylor,et al.  Forecasting Economic Time Series , 1979 .

[7]  Lawrence J. Christiano.  P*: not the inflation forecaster's holy grail , 1989 .

[8]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[9]  D. Hendry,et al.  Econometric Evaluation of Linear Macro-Economic Models , 1986 .

[10]  Kenneth D West,et al.  Tests for Forecasts Encompassing When Forecasts Depend on Estimated Regression Parameters , 2001 .

[11]  Peter Reinhard Hansen,et al.  Model Confidence Sets for Forecasting Models , 2005 .

[12]  Todd E. Clark,et al.  Tests of Equal Forecast Accuracy and Encompassing for Nested Models , 1999 .

[13]  L. Hansen.  Large Sample Properties of Generalized Method of Moments Estimators , 1982 .

[14]  P. Burridge,et al.  A Very Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix , 1991 .

[15]  Michael W. McCracken.  Robust out-of-sample inference , 2000 .

[16]  Gordon Leitch,et al.  Economic Forecast Evaluation: Profits versus the Conventional Error Measures , 1991 .

[17]  D. Thornton,et al.  Federal Funds Rate Prediction , 2004 .

[18]  H. White.  Asymptotic theory for econometricians , 1985 .

[19]  Yin-Wong Cheung,et al.  Empirical Exchange Rate Models of the Nineties: Are Any Fit to Survive? , 2002 .

[20]  Michael P. Clements,et al.  A comparison of tests of nonlinear cointegration with application to the predictability of US interest rates using the term structure , 2004 .

[21]  Adrian Pagan,et al.  Diagnostic tests as residual analysis , 1983 .

[22]  Norman R. Swanson,et al.  Nonparametric Bootstrap Procedures for Predictive Inference Based on Recursive Estimation Schemes , 2005 .

[23]  Allan Timmermann,et al.  Optimal Forecast Combinations Under General Loss Functions and Forecast Error Distributions , 2002 .

[24]  Todd E. Clark,et al.  Evaluating Long – Horizon Forecasts ∗ , 2004 .

[25]  K. West,et al.  The Predictive Ability of Several Models of Exchange Rate Volatility , 1994 .

[26]  Doron Avramov,et al.  Stock Return Predictability and Model Uncertainty , 2001 .

[27]  Jonathan H. Wright,et al.  Forecasting Inflation , 2011 .

[28]  Donald W. K. Andrews,et al.  An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator , 1992 .

[29]  L. Kilian,et al.  In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use? , 2002, SSRN Electronic Journal.

[30]  Richard Schmalensee,et al.  Advertising and aggregate consumption: an analysis of causality , 1980 .

[31]  K. West,et al.  A Utility Based Comparison of Some Models of Exchange Rate Volatility , 1992 .

[32]  Michael W. McCracken,et al.  Evaluating the Predictability of Exchange Rates Using Long-Horizon Regressions: Mind Your p 's and q 's! , 2005 .

[33]  W. Newey,et al.  Automatic Lag Selection in Covariance Matrix Estimation , 1994 .

[34]  Barbara Rossi,et al.  Testing Long-Horizon Predictive Ability with High Persistence, and the Meese-Rogoff Puzzle , 2005 .

[35]  Jonathan H. Wright,et al.  Federal Reserve Board , 2000 .

[36]  Todd E. Clark,et al.  Approximately Normal Tests for Equal Predictive Accuracy in Nested Models , 2005 .

[37]  Norman R. Swanson,et al.  Predictive Density Evaluation , 2005 .

[38]  C. Hueng.  Money demand in an open-economy shopping-time model: an out-of-sample-prediction application to Canada , 1999 .

[39]  C. Hueng,et al.  Predictive Abilities of Inflation-Forecasting Models Using Real Time Data , 2000 .

[40]  Michael Wolf,et al.  Centre De Referència En Economia Analítica Barcelona Economics Working Paper Series Working Paper Nº 17 Stewise Multiple Testing as Formalized Data Snooping Stepwise Multiple Testing as Formalized Data Snooping , 2022 .

[41]  W. A. Morgan.  TEST FOR THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN THE TWO VARIANCES IN A SAMPLE FROM A NORMAL BIVARIATE POPULATION , 1939 .

[42]  H. White,et al.  A Reality Check for Data Snooping , 2000 .

[43]  Lutz Kilian,et al.  On the Selection of Forecasting Models , 2003, SSRN Electronic Journal.

[44]  Kenneth S. Rogoff,et al.  Exchange rate models of the seventies. Do they fit out of sample , 1983 .

[45]  E. Wilson.  The Periodogram of American Business Activity , 1934 .

[46]  Norman R. Swanson,et al.  Predictive ability with cointegrated variables , 2001 .

[47]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[48]  Todd E. Clark,et al.  The power of tests of predictive ability in the presence of structural breaks , 2005 .

[49]  W. D. Haan.  Evaluation and Combination of Conditional Quantile Forecasts , 2000 .

[50]  D. Andrews.  Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation , 1991 .

[51]  Eric Ghysels,et al.  A Test for Structural Stability of Euler Conditions Parameters Estimated Via the Generalized Methods of Moments Estimators , 1990 .

[52]  N. Mark,et al.  Exchange Rates and Fundamentals: Evidence on Long-Horizon Predictability , 1995 .

[53]  Tae-Hwy Lee,et al.  Inference on Predictability of Foreign Exchange Rates via Generalized Spectrum and Nonlinear Time Series Models , 2004, Review of Economics and Statistics.

[54]  J. Stock,et al.  Macroeconomic Forecasting Using Diffusion Indexes , 2002 .

[55]  P. Hansen.  A Test for Superior Predictive Ability , 2005 .

[56]  Michael W. McCracken.  Asymptotics for Out of Sample Tests of Causality , 1998 .

[57]  P. Newbold,et al.  Tests for Forecast Encompassing , 1998 .

[58]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[59]  Miguel A. Ferreira.  Forecasting the Comovements of Spot Interest Rates , 2000 .

[60]  Dennis L. Hoffman,et al.  PRACTITIONERS CORNER: Post‐Sample Prediction Tests for Generalized Method of Moments Estimators , 2009 .

[61]  Halbert White,et al.  Tests of Conditional Predictive Ability , 2003 .

[62]  James G. MacKinnon,et al.  Model Specification Tests Based on Artificial Linear Regressions , 1984 .

[63]  Todd E. Clark,et al.  Using Out-of-Sample Mean Squared Prediction Errors to Test the Martingale Difference Hypothesis , 2004 .

[64]  J. Wooldridge.  A Unified Approach to Robust, Regression-Based Specification Tests , 1990, Econometric Theory.

[65]  Joseph P. Romano,et al.  The stationary bootstrap , 1994 .