The Model Confidence Set

The paper introduces the model confidence set (MCS) and applies it to the selection of models. A MCS is a set of models that is constructed such that it will contain the best model with a given level of confidence. The MCS is in this sense analogous to a confidence interval for a parameter. The MCS acknowledges the limitations of the data, such that uninformative data yields a MCS with many models, whereas informative data yields a MCS with only a few models. The MCS procedure does not assume that a particular model is the true model, in fact the MCS procedure can be used to comparemore general objects, beyond the comparison of models. We apply the MCS procedure to two empirical problems. First, we revisit the inflation forecasting problem posed by Stock and Watson (1999), and compute the MCS for their set of inflation forecasts. Second, we compare a number of Taylor rule regressions and determine the MCS of the best in terms of in-sample likelihood criteria.

[1]  J. Galí,et al.  Monetary Policy Rules and Macroeconomic Stability: Evidence and Some Theory , 1998 .

[2]  Todd E. Clark.  Evaluating Direct Multi – Step Forecasts * , 2005 .

[3]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[4]  J. Stock,et al.  Forecasting Output and Inflation: The Role of Asset Prices , 2001 .

[5]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[6]  Hidetoshi Shimodaira.  An Application of Multiple Comparison Techniques to Model Selection , 1998 .

[7]  Jonathan H. Wright,et al.  Forecasting Inflation , 2011 .

[8]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[9]  James M. Nason,et al.  Identifying the New Keynesian Phillips Curve , 2005 .

[10]  A. Atkeson,et al.  Are Phillips Curves Useful for Forecasting Inflation , 2001 .

[11]  J. Stock,et al.  The NAIRU, Unemployment and Monetary Policy , 1997 .

[12]  Hannes Leeb,et al.  The Finite-Sample Distribution of Post-Model-Selection Estimators, and Uniform Versus Non-Uniform Approximations , 2000 .

[13]  Joseph P. Romano,et al.  The stationary bootstrap , 1994 .

[14]  S. Dudoit,et al.  Multiple Hypothesis Testing in Microarray Experiments , 2003 .

[15]  Jean Boivin,et al.  Monetary Policy in a Data-Rich Environment , 2001 .

[16]  Norman R. Swanson,et al.  An Out of Sample Test for Granger Causality , 2000 .

[17]  S. Johansen.  STATISTICAL ANALYSIS OF COINTEGRATION VECTORS , 1988 .

[18]  Peter Schmidt,et al.  Multiple comparisons with the best, with economic applications , 2000 .

[19]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[20]  A. Harvey,et al.  General Model-Based Filters for Extracting Cycles and Trends in Economic Time Series , 2003, Review of Economics and Statistics.

[21]  E. Prescott,et al.  Postwar U.S. Business Cycles: An Empirical Investigation , 1997 .

[22]  Halbert White,et al.  Estimation, inference, and specification analysis , 1996 .

[23]  J. Cavanaugh,et al.  A BOOTSTRAP VARIANT OF AIC FOR STATE-SPACE MODEL SELECTION , 1997 .

[24]  Morris H. Degroot.  Multiple Decision Procedures , 2006 .

[25]  John B. Taylor,et al.  An Historical Analysis of Monetary Policy Rules , 1998 .

[26]  D. Hendry,et al.  Econometric Evaluation of Linear Macro-Economic Models , 1986 .

[27]  Lutz Kilian,et al.  Exchange Rates and Monetary Fundamentals: What Do We Learn from Long-Horizon Regressions? , 1999 .

[28]  Roger L. Berger,et al.  Multiparameter Hypothesis Testing and Acceptance Sampling , 1982 .

[29]  Sastry G. Pantula,et al.  Testing for Unit Roots in Time Series Data , 1989, Econometric Theory.

[30]  Azeem M. Shaikh,et al.  FORMALIZED DATA SNOOPING BASED ON GENERALIZED ERROR RATES , 2007, Econometric Theory.

[31]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[32]  Ken West.  Asymptotic Inference about Predictive Ability, An Additional Appendix , 1994 .

[33]  T. W. Anderson.  An Introduction to Multivariate Statistical Analysis , 1959 .

[34]  Martin M. Andreasen.  How Non-Gaussian Shocks Affect Risk Premia in Non-Linear DSGE Models , 2011 .

[35]  Athanasios Orphanides.  Historical monetary policy analysis and the Taylor rule , 2003 .

[36]  Jurgen A. Doornik,et al.  An object-orientd matrix programming language Ox 4 , 2006 .

[37]  B. Mccallum,et al.  Issues in the Design of Monetary Policy Rules , 1997 .

[38]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[39]  John B. Taylor.  Discretion versus policy rules in practice , 1993 .

[40]  J. Stock,et al.  How Precise are Estimates of the Natural Rate of Unemployment? , 1996 .

[41]  D. Andrews.  Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation , 1991 .

[42]  Halbert White,et al.  Tests of Conditional Predictive Ability , 2003 .

[43]  Lutz Kilian,et al.  On the Selection of Forecasting Models , 2003, SSRN Electronic Journal.

[44]  Michael Wolf,et al.  Centre De Referència En Economia Analítica Barcelona Economics Working Paper Series Working Paper Nº 17 Stewise Multiple Testing as Formalized Data Snooping Stepwise Multiple Testing as Formalized Data Snooping , 2022 .

[45]  Peter Reinhard Hansen,et al.  Asymptotic Tests of Composite Hypotheses , 2003 .

[46]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[47]  H. White,et al.  Information criteria for selecting possibly misspecified parametric models , 1996 .

[48]  P. Hansen.  A Test for Superior Predictive Ability , 2005 .

[49]  Han Hong,et al.  Bayesian Averaging, Prediction and Nonnested Model Selection , 2008 .

[50]  Athanasios Orphanides,et al.  The Unreliability of Output-Gap Estimates in Real Time , 2002, Review of Economics and Statistics.

[51]  K. West,et al.  The Predictive Ability of Several Models of Exchange Rate Volatility , 1994 .

[52]  Paul Newbold,et al.  Tests for multiple forecast encompassing , 2000 .

[53]  B. Efron.  How Biased is the Apparent Error Rate of a Prediction Rule , 1986 .

[54]  Michael W. McCracken,et al.  Evaluating the Predictability of Exchange Rates Using Long-Horizon Regressions: Mind Your p 's and q 's! , 2005 .

[55]  Peter Reinhard Hansen,et al.  Regression Analysis with Many Specifications: A Bootstrap Method for Robust Inference , 2003 .

[56]  Marianne Baxter,et al.  Measuring Business Cycles: Approximate Band-Pass Filters for Economic Time Series , 1995, Review of Economics and Statistics.

[57]  Jordi Galí,et al.  Monetary Policy Rules and Macroeconomic Stability: Evidence and Some Theory , 1998 .

[58]  E. Lehmann.  Testing Statistical Hypotheses , 1960 .

[59]  R. Shibata.  BOOTSTRAP ESTIMATE OF KULLBACK-LEIBLER INFORMATION FOR MODEL SELECTION , 1997 .

[60]  Athanasios Orphanides,et al.  Historical Monetary Policy Analysis and the Taylor Rule , 2003 .

[61]  Todd E. Clark,et al.  Tests of Equal Forecast Accuracy and Encompassing for Nested Models , 1999 .

[62]  Robert F. Engle,et al.  Model selection for forecasting , 1986 .

[63]  Sílvia Gonçalves,et al.  Bootstrap Standard Error Estimates for Linear Regression , 2005 .

[64]  Todd E. Clark,et al.  Tests of Equal Forecast Accuracy and Encompassing for Nested Models , 1999 .

[65]  Clive W. J. Granger,et al.  Comments on testing economic theories and the use of model selection criteria , 1995 .

[66]  Selectionfrom Statistica Sinica.  A Bootstrap Variant of AICfor State-Space Model , 1997 .

[67]  H. White,et al.  A Reality Check for Data Snooping , 2000 .