Comparing Density Forecasts via Weighted Likelihood Ratio Tests

This paper proposes and analyzes tests that can be used to compare the accuracy of alternative conditional density forecasts of a variable. The tests are also valid in the broader context of model selection based on out-of-sample predictive ability. We restrict attention to the case of density forecasts derived from non-nested parametric models, with known or estimated parameters. The evaluation makes use of scoring rules, which are loss functions defined over the density forecast and the realizations of the variable. In particular, we consider the logarithmic scoring rule, which leads to the development of asymptotic and bootstrap 'weighted likelihood ratio' tests. The name comes from the fact that the tests compare weighted averages of the scores over the available sample, as a way to focus attention on different regions of the distribution of the variable. For a uniform weight function, the asymptotic test can be interpreted as an extension of Vuong (1989)' s likelihood ratio test for non-nested hypotheses to time series data and to an out-of-sample testing framework. A Monte Carlo simulation explores the size and power properties of this last test in finite samples. An application using S&P500 daily returns shows how the tests can be used to compare the performance of density forecasts obtained from GARCH models with different distributional assumptions.

[1]  Daniel B. Nelson.  CONDITIONAL HETEROSKEDASTICITY IN ASSET RETURNS: A NEW APPROACH , 1991 .

[2]  Anthony S. Tay,et al.  Evaluating Density Forecasts with Applications to Financial Risk Management , 1998 .

[3]  R. Fisher,et al.  Statistical Methods for Research Workers , 1930, Nature.

[4]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[5]  K. West,et al.  Asymptotic Inference about Predictive Ability , 1996 .

[6]  Andrew A. Weiss,et al.  Estimating Time Series Models Using the Relevant Cost Function , 1996 .

[7]  F. Eugene.  FAMA, . The Behavior of Stock-Market Prices, Journal of Business, , . , 1965 .

[8]  J. Shao,et al.  The jackknife and bootstrap , 1996 .

[9]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[10]  D. Duffie,et al.  An Overview of Value at Risk , 1997 .

[11]  Tim Bollerslev,et al.  Chapter 49 Arch models , 1994 .

[12]  B. Mandelbrot.  The Variation of Certain Speculative Prices , 1963 .

[13]  D. Andrews.  Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation , 1991 .

[14]  F. Diebold,et al.  Further results on forecasting and model selection under asymmetric loss , 1996 .

[15]  Lars E. O. Svensson,et al.  New Techniques to Extract Market Expectations from Financial Instruments , 1997 .

[16]  E. Fama.  The Behavior of Stock-Market Prices , 1965 .

[17]  Anthony S. Tay,et al.  Evaluating Density Forecasts of Inflation: The Survey of Professional Forecasters , 1997 .

[18]  Clive W. J. Granger,et al.  A Decision_Theoretic Approach to Forecast Evaluation , 2000 .

[19]  Michael P. Clements,et al.  Evaluating The Forecast of Densities of Linear and Non-Linear Models: Applications to Output Growth and Unemployment , 2000 .

[20]  Regina Y. Liu.  Moving blocks jackknife and bootstrap capture weak dependence , 1992 .

[21]  T. Bollerslev,et al.  A CONDITIONALLY HETEROSKEDASTIC TIME SERIES MODEL FOR SPECULATIVE PRICES AND RATES OF RETURN , 1987 .

[22]  Freddy Delbaen,et al.  A Characterization of Measures of Risk , 1997 .

[23]  H. White,et al.  A Reality Check for Data Snooping , 2000 .

[24]  F. Diebold,et al.  Optimal Prediction Under Asymmetric Loss , 1994, Econometric Theory.

[25]  Kenneth F. Wallis,et al.  Density Forecasting: A Survey , 2000 .

[26]  Andrew J. Patton.  Modelling Time-Varying Exchange Rate Dependence Using the Conditional Copula , 2001 .

[27]  F. Diebold,et al.  Forecast Evaluation and Combination , 1996 .

[28]  Jose A. Lopez.  Evaluating the Predictive Accuracy of Volatility Models , 2001 .

[29]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[30]  H. Künsch.  The Jackknife and the Bootstrap for General Stationary Observations , 1989 .

[31]  B. Hansen.  Autoregressive Conditional Density Estimation , 1994 .

[32]  Andreas S. Weigend,et al.  Predicting Daily Probability Distributions of S&P500 Returns , 1998 .

[33]  W. Newey,et al.  A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelationconsistent Covariance Matrix , 1986 .

[34]  Michael W. McCracken.  Robust out-of-sample inference , 2000 .

[35]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[36]  H. White,et al.  Nonlinear Regression with Dependent Observations , 1984 .

[37]  Joseph P. Romano,et al.  The stationary bootstrap , 1994 .

[38]  D. Rivers,et al.  Model Selection Tests for Nonlinear Dynamic Models , 2002 .