The role of prediction in evaluating econometric models

Many statistical and economic criteria must influence the overall evaluation of econometric systems, but success when ‘predictive testing’ seems to establish most credibility, perhaps because the data were unavailable to the modellers. Yet the ability of prediction tests to detect even gross mis-specifications depends both on the properties of the data process and on the structure of the selected tests. To study the issues involved, a mixed analytical-Monte Carlo approach was adopted, based on asymptotically valid approximations to forecast confidence intervals. Tests were calibrated for finite samples by simulation experiments on a Distributed Array Processor to yield numerical power function response surfaces. The main results of analysing prediction tests are that H successive one-step ahead tests are equivalent to one H-step test; that conditional multi-step forecast confidence bands are not necessarily monotonically increasing in H, so that ‘intercept corrections’ potentially have an objective justification; that tests based on averages (e. g. within-years) may be more useful than either one-step or H-step tests for detecting predictive failure; that the need to pool forecasts indicates non-encompassing models; and that a forecast-encompassing test should have some diagnostic power when applied to macroeconometric systems. Forecasting of the level of uncertainty is briefly considered.