Can Out-of-Sample Forecast Comparisons Help Prevent Overfitting?

This paper shows that out-of-sample forecast comparisons can help prevent data mining-induced overfitting. The basic results are drawn from simulations of a simple Monte Carlo design and a real data-based design similar to those used in some previous studies. In each simulation, a general-to-specific procedure is used to arrive at a model. If the selected specification includes any of the candidate explanatory variables, forecasts from the model are compared to forecasts from a benchmark model that is nested within the selected model. In particular, the competing forecasts are tested for equal MSE and encompassing. The simulations indicate most of the post-sample tests are roughly correctly sized. Moreover, the tests have relatively good power, although some are consistently more powerful than others. The paper concludes with an application, modelling quarterly US inflation. Copyright © 2004 John Wiley & Sons, Ltd.

[1]  Clive W. J. Granger,et al.  Comments on testing economic theories and the use of model selection criteria , 1995 .

[2]  Thomas Knox,et al.  Empirical Bayes Forecasts of One Time Series Using Many Predictors , 2001 .

[3]  Grayham E. Mizon,et al.  Progressive Modeling of Macroeconomic Time Series The LSE Methodology , 1995 .

[4]  Bruce E. Hansen,et al.  Discussion of 'Data mining reconsidered' , 1999 .

[5]  Michael W. McCracken.  Asymptotics for Out of Sample Tests of Causality , 1998 .

[6]  P. Newbold,et al.  Tests for Forecast Encompassing , 1998 .

[7]  David J. Hand.  Discussion contribution on ‘Data mining reconsidered: encompassing and the general‐to‐specific approach to specification search’ by Hoover and Perez , 1999 .

[8]  C. Granger,et al.  Forecasting Economic Time Series. , 1988 .

[9]  Richard Schmalensee,et al.  Advertising and aggregate consumption: an analysis of causality , 1980 .

[10]  Sydney C. Ludvigson,et al.  Consumption, Aggregate Wealth and Expected Stock Returns , 1999 .

[11]  Ian Witten,et al.  Data Mining , 2000 .

[12]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[13]  Menzie David Chinn,et al.  Banking on currency forecasts: How predictable is change in money? , 1995 .

[14]  H. Theil.  Principles of econometrics , 1971 .

[15]  J. Stock,et al.  Diffusion Indexes , 1998 .

[16]  Martin D. D. Evans,et al.  Order Flow and Exchange Rate Dynamics , 2002, Journal of Political Economy.

[17]  Kenneth S. Rogoff,et al.  Exchange rate models of the seventies. Do they fit out of sample , 1983 .

[18]  David F. Hendry,et al.  The Demand for M1 in the U.S.A., 1960–1988 , 1992 .

[19]  Timothy Cogley,et al.  A Simple Adaptive Measure of Core Inflation , 2002 .

[20]  Todd E. Clark,et al.  Tests of Equal Forecast Accuracy and Encompassing for Nested Models , 1999 .

[21]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[22]  Neil R. Ericsson.  Parameter constancy, mean square forecast errors, and measuring forecast performance: an exposition, extensions, and illustration , 1991 .

[23]  H. White,et al.  A Reality Check for Data Snooping , 2000 .

[24]  David F. Hendry,et al.  Improving on "Data mining reconsidered" by K.D. Hoover and S.J. Perez , 1999 .

[25]  Milton H. Marquis,et al.  Federal Reserve Bank of San Francisco , 2004 .

[26]  Kevin D. Hoover,et al.  Data mining reconsidered: encompassing and the general-to-specific approach to specification search , 1997 .

[27]  Norman R. Swanson,et al.  An Out of Sample Test for Granger Causality , 2000 .

[28]  Norman R. Swanson,et al.  OUT-OF-SAMPLE TESTS FOR GRANGER CAUSALITY , 2001, Macroeconomic Dynamics.

[29]  Kenneth Rogoff,et al.  Was It Real? The Exchange Rate‐Interest Differential Relation over the Modern Floating‐Rate Period , 1988 .

[30]  W. J. Martin,et al.  The terms of trade and real exchange rates , 1989 .

[31]  David F. Hendry,et al.  Computer Automation of General-to-Specific Model Selection Procedures , 2001 .

[32]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[33]  Simon van Norden,et al.  Terms of trade and real exchange rates: the Canadian evidence , 1995 .

[34]  Neil R. Ericsson,et al.  Constructive data mining: modeling consumers' expenditure in Venezuela , 1999 .

[35]  K. Hoover,et al.  Improving on ‘ Data mining reconsidered ’ by , 2000 .

[36]  Karl Rihaczek,et al.  1. WHAT IS DATA MINING? , 2019, Data Mining for the Social Sciences.

[37]  Martin D. D. Evans,et al.  Order Flow and Exchange Rate Dynamics , 1999, Journal of Political Economy.