Bagging binary and quantile predictors for time series

Abstract Bootstrap aggregating or Bagging, introduced by Breiman (1996a. Bagging predictors. Machine Learning 24, 123–140), has been proved to be effective to improve on unstable forecast. Theoretical and empirical works using classification, regression trees, variable selection in linear and non-linear regression have shown that bagging can generate substantial prediction gain. However, most of the existing literature on bagging has been limited to the cross sectional circumstances with symmetric cost functions. In this paper, we extend the application of bagging to time series settings with asymmetric cost functions, particularly for predicting signs and quantiles. We use quantile predictions to construct a binary predictor and the majority-voted bagging binary prediction. We show that bagging may improve the binary prediction in small sample, but it does not improve in large sample. Various bagging forecast combination weights are used such as equal weighted and Bayesian model averaging (BMA) weighted combinations. For demonstration, we present results from Monte Carlo experiments and from empirical applications using monthly S&P500 and NASDAQ stock index returns.

[1]  L. Breiman.  Heuristics of instability and stabilization in model selection , 1996 .

[2]  Allan Timmermann,et al.  Estimating Loss Function Parameters , 2003 .

[3]  Tae-Hwan Kim,et al.  Estimation, Inference, and Specification Testing for Possibly Misspecified Quantile Regression , 2002 .

[4]  Clive W. J. Granger,et al.  Some comments on risk , 2002 .

[5]  R. Koenker,et al.  An interior point algorithm for nonlinear quantile regression , 1996 .

[6]  Clive W. J. Granger,et al.  Outline of forecast theory using generalized cost functions , 1999 .

[7]  C. Manski.  Semiparametric analysis of discrete response: Asymptotic properties of the maximum score estimator , 1985 .

[8]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[9]  Francis X. Diebold,et al.  The Rodney L. White Center for Financial Research Financial Asset Returns, Direction-of-Change Forecasting and Volatility , 2003 .

[10]  David F. Hendry,et al.  Non-Parametric Direct Multi-Step Estimation for Forecasting Economic Processes , 2004 .

[11]  C. Granger,et al.  Handbook of Economic Forecasting , 2006 .

[12]  J. Friedman,et al.  On bagging and nonlinear estimation , 2007 .

[13]  Stephen Portnoy,et al.  Censored Regression Quantiles , 2003 .

[14]  R. Koenker,et al.  Asymptotic Theory of Least Absolute Error Regression , 1978 .

[15]  V. Chernozhukov,et al.  An MCMC approach to classical estimation , 2003 .

[16]  Ruey S. Tsay,et al.  Co‐integration constraint and forecasting: An empirical examination , 1996 .

[17]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[18]  Moshe Buchinsky.  Recent Advances in Quantile Regression Models: A Practical Guideline for Empirical Research , 1998 .

[19]  J. B. G. Frenk,et al.  A deep cut ellipsoid algorithm for convex programming: Theory and applications , 1994, Math. Program..

[20]  Gregory Kordas.  Smoothed binary regression quantiles , 2006 .

[21]  R. Koenker,et al.  The Gaussian hare and the Laplacian tortoise: computability of squared-error versus absolute-error estimators , 1997 .

[22]  R. Mariano,et al.  Residual-Based Procedures for Prediction and Estimation in a Nonlinear Simultaneous System , 1984 .

[23]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[24]  C. Granger,et al.  Economic and Statistical Measures of Forecast Accuracy , 1999 .

[25]  A. Buja,et al.  OBSERVATIONS ON BAGGING , 2006 .

[26]  C. Manski.  MAXIMUM SCORE ESTIMATION OF THE STOCHASTIC UTILITY MODEL OF CHOICE , 1975 .

[27]  Yuhong Yang.  COMBINING FORECASTING PROCEDURES: SOME THEORETICAL RESULTS , 2004, Econometric Theory.

[28]  H. White.  Nonparametric Estimation of Conditional Quantiles Using Neural Networks , 1990 .

[29]  L. Kilian,et al.  How Useful Is Bagging in Forecasting Economic Time Series? A Case Study of U.S. Consumer Price Inflation , 2008 .

[30]  D. Pollard,et al.  Cube Root Asymptotics , 1990 .

[31]  Lutz Kilian,et al.  How Useful is Bagging in Forecasting Economic Time Series? A Case Study of Us CPI Inflation , 2005 .

[32]  Doron Avramov,et al.  Stock Return Predictability and Model Uncertainty , 2001 .

[33]  O. Linton,et al.  A Quantilogram Approach to Evaluating Directional Predictability , 2003 .

[34]  Victor Chernozhukov,et al.  Conditional value-at-risk: Aspects of modeling and estimation , 2000 .

[35]  Tae-Hwy Lee,et al.  Inference on Predictability of Foreign Exchange Rates via Generalized Spectrum and Nonlinear Time Series Models , 2003, Review of Economics and Statistics.

[36]  Anthony Garratt,et al.  Forecast Uncertainties in Macroeconomic Modeling , 2003 .

[37]  Ruey S. Tsay,et al.  Comment: Adaptive Forecasting , 1993 .

[38]  J. Stock,et al.  A Comparison of Linear and Nonlinear Univariate Models for Forecasting Macroeconomic Time Series , 1998 .

[39]  M. Hashem Pesaran,et al.  How Costly is it to Ignore Breaks When Forecasting the Direction of a Time Series? , 2003, SSRN Electronic Journal.

[40]  H. Nyquist.  The optimal Lp norm estimator in linear regression models , 1983 .

[41]  Tae-Hwy Lee,et al.  Inference on Predictability of Foreign Exchange Rates via Generalized Spectrum and Nonlinear Time Series Models , 2004, Review of Economics and Statistics.

[42]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[43]  A. Timmermann.  Forecast Combinations , 2005 .

[44]  Allan Timmermann,et al.  Biases in Macroeconomic Forecasts: Irrationality or Asymmetric Loss? , 2008 .

[45]  Anthony Garratt,et al.  Forecast Uncertainties in Macroeconometric Modelling: An Application to the UK Economy , 2000, SSRN Electronic Journal.

[46]  Herbert K. H. Lee.  Consistency of posterior distributions for neural networks , 2000, Neural Networks.

[47]  C. Manski,et al.  Estimation of best predictors of binary response , 1989 .

[48]  Mark W. Watson,et al.  AN EMPIRICAL COMPARISON OF METHODS FOR FORECASTING USING MANY PREDICTORS , 2005 .

[49]  Halbert White,et al.  Estimation, inference, and specification analysis , 1996 .

[50]  Clive W. J. Granger,et al.  Prediction with a generalized cost of error function , 1969 .

[51]  A. Timmermann,et al.  Market timing and return prediction under model instability , 2002 .

[52]  Bernd Fitzenberger,et al.  The moving blocks bootstrap and robust inference for linear least squares and quantile regressions , 1998 .

[53]  Hui Zou,et al.  Combining time series models for forecasting , 2004, International Journal of Forecasting.

[54]  H. Leon Harter,et al.  Nonuniqueness of least absolute values regression , 1977 .

[55]  A. Money,et al.  The linear regression model: Lp norm estimation and the choice of p , 1982 .

[56]  P. Bühlmann,et al.  Analyzing Bagging , 2001 .

[57]  H. Chipman,et al.  Bayesian CART Model Search , 1998 .

[58]  P. Hall.  The Bootstrap and Edgeworth Expansion , 1992 .

[59]  Ivana Komunjer,et al.  Quasi-maximum likelihood estimation for conditional quantiles , 2005 .

[60]  W. Newey,et al.  Large sample estimation and hypothesis testing , 1986 .

[61]  A. Timmermann,et al.  Model Instability and Choice of Observation Window , 1999 .

[62]  Massimiliano Pontil,et al.  Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers , 2004, Machine Learning.

[63]  Timo Teräsvirta,et al.  The combination of forecasts using changing weights , 1994 .

[64]  C. Granger,et al.  Forecasting from non-linear models in practice , 1994 .

[65]  M. Wand,et al.  EXACT MEAN INTEGRATED SQUARED ERROR , 1992 .

[66]  Clive W. J. Granger,et al.  Empirical Modeling in Economics: Specification and Evaluation , 1999 .