Path Forecast Evaluation

A path forecast refers to the sequence of forecasts 1 to H periods into the future. A summary of the range of possible paths the predicted variable may follow for a given confidence level requires construction of simultaneous confidence regions that adjust for any covariance between the elements of the path forecast. This paper shows how to construct such regions with the joint predictive density and Scheffe's (1953) S-method. In addition, the joint predictive density can be used to construct simple statistics to evaluate the local internal consistency of a forecasting exercise of a system of variables. Monte Carlo simulations demonstrate that these simultaneous confidence regions provide approximately correct coverage in situations where traditional error bands, based on the collection of marginal predictive densities for each horizon, are vastly off mark. The paper showcases these methods with an application to the most recent monetary episode of interest rate hikes in the U.S. macroeconomy.

[1]  Norman R. Swanson,et al.  Predictive ability with cointegrated variables , 2001 .

[2]  James D. Hamilton.  Time Series Analysis , 1994 .

[3]  R. J. Bhansali,et al.  Multi‐Step Forecasting , 2007 .

[4]  A. Greenspan.  Closing remarks: a speech at a symposium sponsored by the Federal Reserve Bank of Kansas City, Jackson Hole, Wyoming, September 9, 2005 , 2005 .

[5]  S. N. Roy.  On a Heuristic Method of Test Construction and its use in Multivariate Analysis , 1953 .

[6]  Anthony Garratt,et al.  Forecast Uncertainties in Macroeconomic Modeling , 2003 .

[7]  H. Scheffé.  A METHOD FOR JUDGING ALL CONTRASTS IN THE ANALYSIS OF VARIANCE , 1953 .

[8]  J. Ringland.  Robust Multiple Comparisons , 1983 .

[9]  R. Lucas.  Econometric policy evaluation: A critique , 1976 .

[10]  Michael W. McCracken.  Robust out-of-sample inference , 2000 .

[11]  A. Tamhane,et al.  Multiple Comparison Procedures , 2009 .

[12]  T. W. Anderson.  The Statistical Analysis of Time Series: Anderson/The Statistical , 1994 .

[13]  Clifford M. Hurvich,et al.  A CORRECTED AKAIKE INFORMATION CRITERION FOR VECTOR AUTOREGRESSIVE MODEL SELECTION , 1993 .

[14]  G. Reinsel,et al.  Prediction of multivariate time series by autoregressive model fitting , 1985 .

[15]  Helmut Ltkepohl,et al.  New Introduction to Multiple Time Series Analysis , 2007 .

[16]  Thomas S. Ferguson,et al.  A Method of Generating Best Asymptotically Normal Estimates with Application to the Estimation of Bacterial Densities , 1958 .

[17]  N. E. Savin,et al.  The Bonferroni and the Scheffé multiple comparison procedures , 1980 .

[18]  Gary Chamberlain,et al.  Analysis of Covariance with Qualitative Data , 1979 .

[19]  Jean-Marie Dufour,et al.  Nonlinear hypotheses, inequality restrictions, and non-nested hypotheses: exact simultaneous tests in linear regressions , 1989 .

[20]  A. Cameron,et al.  Microeconometrics: Methods and Applications , 2005 .

[21]  Ò. Jordà,et al.  Estimation and Inference of Impulse Responses by Local Projections , 2005 .

[22]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[23]  D. C. Bowden,et al.  Simultaneous Confidence Bands for Linear Regression Models , 1970 .

[24]  S. Kozicki,et al.  Estimation and Inference by the Method of Projection Minimum Distance , 2007 .

[25]  J. Stock,et al.  A Comparison of Direct and Iterated Multistep Ar Methods for Forecasting Macroeconomic Time Series , 2005 .

[26]  R. Baillie,et al.  Prediction in dynamic models with time-dependent conditional variances , 1992 .

[27]  P. Mahalanobis.  On the generalized distance in statistics , 1936 .

[28]  A. F. Mitchell,et al.  The Mahalanobis distance and elliptic distributions , 1985 .

[29]  H. White.  A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity , 1980 .

[30]  Halbert White,et al.  Tests of Conditional Predictive Ability , 2003 .

[31]  Jean-Marie Dufour,et al.  Exact tests and confidence sets in linear regressions with autocorrelated errors , 1990 .

[32]  R. C. Bose,et al.  Simultaneous Confidence Interval Estimation , 1953 .

[33]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[34]  M. Hashem Pesaran,et al.  Selection of estimation window in the presence of breaks , 2007 .

[35]  J. Stock,et al.  Macroeconomic forecasting in the Euro area: Country specific versus area-wide information , 2003 .

[36]  T. W. Anderson,et al.  Statistical analysis of time series , 1972 .

[37]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[38]  Optimal Instrumental Variables Estimation For ARMA Models , 2001 .

[39]  Daniel F. Waggoner,et al.  Conditional Forecasts in Dynamic Multivariate Models , 1998, Review of Economics and Statistics.

[40]  Lutz Kilian,et al.  Asymptotic and Bootstrap Inference for AR(∞) Processes with Conditional Heteroskedasticity , 2007 .

[41]  E. Lehmann,et al.  Testing Statistical Hypothesis. , 1960 .

[42]  Michael P. Clements,et al.  On the limitations of comparing mean square forecast errors , 1993 .

[43]  Norman R. Swanson,et al.  An Out of Sample Test for Granger Causality , 2000 .

[44]  W. C. Guenther,et al.  Analysis of variance , 1968, The Mathematical Gazette.

[45]  Todd E. Clark,et al.  Tests of Equal Forecast Accuracy and Encompassing for Nested Models , 1999 .

[46]  A. Wald.  Tests of statistical hypotheses concerning several parameters when the number of observations is large , 1943 .

[47]  Anthony Garratt,et al.  Forecast Uncertainties in Macroeconometric Modelling: An Application to the UK Economy , 2000, SSRN Electronic Journal.

[48]  Ò. Jordà,et al.  Simultaneous Confidence Regions for Impulse Responses , 2009 .

[49]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[50]  Guido M. Kuersteiner,et al.  Efficiency IV Estimation for Autoregressive Models with Conditional Heterogeneity , 2000 .

[51]  Lutz Kilian,et al.  NEW INTRODUCTION TO MULTIPLE TIME SERIES ANALYSIS, by Helmut Lütkepohl, Springer, 2005 , 2006, Econometric Theory.

[52]  Helmut Lütkepohl,et al.  Estimating Orthogonal Impulse Responses via Vector Autoregressive Models , 1991, Econometric Theory.

[53]  A. Tamhane,et al.  Multiple Comparison Procedures , 1989 .

[54]  Joel L. Horowitz,et al.  The bootstrap and hypothesis tests in econometrics , 2001 .

[55]  N. Savin.  MULTIPLE HYPOTHESIS TESTING , 1984 .

[56]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[57]  Jean-Marie Dufour,et al.  Projection-Based Statistical Inference in Linear Structural Models with Possibly Weak Instruments , 2005 .

[58]  Lutz Kilian,et al.  Frequentist Inference in Weakly Identified DSGE Models , 2009 .

[59]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .