Improving Regressors using Boosting Techniques

In the regression context, boosting and bagging are techniques to build a committee of regressors that may be superior to a single regressor. We use regression trees as fundamental building blocks in bagging committee machines and boosting committee machines. Performance is analyzed on three non-linear functions and the Boston housing database. In all cases, boosting is at least equivalent, and in most cases better than bagging in terms of prediction error.

[1]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[2]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[3]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[4]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[5]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[6]  Harris Drucker,et al.  Boosting Performance in Neural Networks , 1993, Int. J. Pattern Recognit. Artif. Intell..

[7]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[8]  Simon Kasif,et al.  OC1: A Randomized Induction of Oblique Decision Trees , 1993, AAAI.

[9]  Harris Drucker,et al.  Boosting and Other Ensemble Methods , 1994, Neural Computation.

[10]  Simon Kasif,et al.  A System for Induction of Oblique Decision Trees , 1994, J. Artif. Intell. Res..

[11]  Corinna Cortes,et al.  Boosting Decision Trees , 1995, NIPS.

[12]  Paul M. B. Vitányi,et al.  Proceedings of the Second European Conference on Computational Learning Theory , 1995 .

[13]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[14]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[15]  Michael Schlosser,et al.  Non-Linear Decision Trees - NDT , 1996, ICML.

[16]  Leo Breiman,et al.  Bias, Variance , And Arcing Classifiers , 1996 .

[17]  Michael Kearns,et al.  A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates, with Consequences for the Training-Test Split , 1995, Neural Computation.

[18]  James S. Harris,et al.  Probability theory and mathematical statistics , 1998 .