Evaluating, Comparing and Combining Density Forecasts Using the Klic with an Application to the Bank of England and Niesr 'Fan' Charts of Inflation

This paper proposes and analyses the Kullback–Leibler information criterion (KLIC) as a unified statistical tool to evaluate, compare and combine density forecasts. Use of the KLIC is particularly attractive, as well as operationally convenient, given its equivalence with the widely used Berkowitz likelihood ratio test for the evaluation of individual density forecasts that exploits the probability integral transforms. Parallels with the comparison and combination of point forecasts are made. This and related Monte Carlo experiments help draw out properties of combined density forecasts. We illustrate the uses of the KLIC in an application to two widely used published density forecasts for UK inflation, namely the Bank of England and NIESR ‘fan’ charts.

[1]  Greg Tkacz,et al.  A Consistent Bootstrap Test for Conditional Density Functions with Time-Dependent Data , 2000 .

[2]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[3]  H. White.  Asymptotic theory for econometricians , 1985 .

[4]  Paul Newbold,et al.  Testing the equality of prediction mean squared errors , 1997 .

[5]  Anthony Garratt,et al.  Forecast Uncertainties in Macroeconomic Modeling , 2003 .

[6]  Kenneth F. Wallis,et al.  Combining Density and Interval Forecasts: A Modest Proposal , 2005 .

[7]  Kenneth F. Wallis,et al.  Macroeconomic Forecasting: A Survey , 1989 .

[8]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[9]  M. Déqué,et al.  Formulation of gaussian probability forecasts based on model extended‐range integrations , 1994 .

[10]  James Mitchell,et al.  Density Forecast Combination , 2004 .

[11]  William A. Bomberger.  Disagreement as a Measure of Uncertainty , 1996 .

[12]  C. Granger,et al.  Improved methods of combining forecasts , 1984 .

[13]  Christian Genest,et al.  Combining Probability Distributions: A Critique and an Annotated Bibliography , 1986 .

[14]  Michael P. Clements.  Some possible directions for future research , 2003 .

[15]  Stephen Taylor,et al.  Forecasting Economic Time Series , 1979 .

[16]  Anthony S. Tay,et al.  Evaluating Density Forecasts with Applications to Financial Risk Management , 1998 .

[17]  Kaplan,et al.  ‘Combining Probability Distributions from Experts in Risk Analysis’ , 2000, Risk analysis : an official publication of the Society for Risk Analysis.

[18]  Norman R. Swanson,et al.  Predictive Density and Conditional Confidence Interval Accuracy Tests , 2004 .

[19]  Michael P. Clements,et al.  Pooling of Forecasts , 2004 .

[20]  M. Pesaran,et al.  Model Averaging and Value-at-Risk Based Evaluation of Large Multi-Asset Volatility Models for Risk Management , 2004, SSRN Electronic Journal.

[21]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[22]  David R. Anderson,et al.  Model Selection and Multimodel Inference , 2003 .

[23]  K. Wallis,et al.  Combining Point Forecasts: The Simple Average Rules, OK? , 2005 .

[24]  A. Blake.  Forecast Error Bounds By Stochastic Simulation , 1996, National Institute Economic Review.

[25]  Kenneth F. Wallis,et al.  Density Forecasting: A Survey , 2000 .

[26]  C. Granger.  Invited review combining forecasts—twenty years later , 1989 .

[27]  Giorgio Valente,et al.  Comparing the accuracy of density forecasts from competing models , 2002 .

[28]  Anthony Garratt,et al.  Forecast Uncertainties in Macroeconometric Modelling: An Application to the UK Economy , 2000, SSRN Electronic Journal.

[29]  James Mitchell.  Optimal combination of density forecasts , 2004 .

[30]  A. Cohen,et al.  Finite Mixture Distributions , 1982 .

[31]  Daniel S. Wilks,et al.  Smoothing forecast ensembles with fitted probability distributions , 2002 .

[32]  C. Granger,et al.  Interval forecasting. An analysis based upon ARCH-quantile estimators , 1989 .

[33]  H. White,et al.  A Reality Check for Data Snooping , 2000 .

[34]  E. Britton,et al.  The Inflation Report projections: understanding the fan chart , 1997 .

[35]  Michael P. Clements,et al.  Evaluating The Forecast of Densities of Linear and Non-Linear Models: Applications to Output Growth and Unemployment , 2000 .

[36]  Jesús Fernández-Villaverde,et al.  Comparing Dynamic Equilibrium Economies to Data: A Bayesian Approach , 2003 .

[37]  R. L. Winkler.  Combining Probability Distributions from Dependent Information Sources , 1981 .

[38]  James Mitchell.  The National Institute Density Forecasts of Inflation , 2005, National Institute Economic Review.

[39]  Anthony S. Tay,et al.  Evaluating Density Forecasts of Inflation: The Survey of Professional Forecasters , 1997 .

[40]  Kenneth F. Wallis,et al.  An Assessment of Bank of England and National Institute Inflation Forecast Uncertainties , 2004, National Institute Economic Review.

[41]  Jesús Fernández-Villaverde,et al.  Comparing dynamic equilibrium models to data: a Bayesian approach , 2004 .

[42]  Peter A. Morris,et al.  Combining Expert Judgments: A Bayesian Approach , 1977 .

[43]  Greg Tkacz,et al.  A consistent bootstrap test for conditional density functions with time-series data , 2006 .

[44]  Paolo Giordani,et al.  Inflation Forecast Uncertainty , 2001 .

[45]  Jeremy Berkowitz.  Testing Density Forecasts, With Applications to Risk Management , 2001 .

[46]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[47]  Michael P. Clements,et al.  Evaluating the Bank of England Density Forecasts of Inflation , 2004 .

[48]  Raffaella Giacomini,et al.  Comparing Density Forecasts via Weighted Likelihood Ratio Tests , 2002 .

[49]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[50]  C. Granger,et al.  Economic and Statistical Measures of Forecast Accuracy , 1999 .

[51]  Halbert White,et al.  Tests of Conditional Predictive Ability , 2003 .

[52]  M. Weale,et al.  The Performance of National Institute Economic Forecasts , 1996, National Institute Economic Review.

[53]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[54]  A. H. Murphy,et al.  Screening probability forecasts: contrasts between choosing and combining , 1995 .

[55]  Yong Bao,et al.  A Test for Density Forecast Comparison with Applications to Risk Management , 2004 .

[56]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[57]  J. Stock,et al.  Combination forecasts of output growth in a seven-country data set , 2004 .

[58]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[59]  D. Madigan,et al.  Bayesian Model Averaging for Linear Regression Models , 1997 .

[60]  Michael P. Clements.  Evaluating the survey of professional forecasters probability distributions of expected inflation based on derived event probability forecasts , 2006 .