Pruning Adaptive Boosting Ensembles by Means of a Genetic Algorithm

This work analyzes the problem of whether, given a classification ensemble built by Adaboost, it is possible to find a subensemble with lower generalization error. In order to solve this task a genetic algorithm is proposed and compared with other heuristics like Kappa pruning and Reduce-error pruning with backfitting. Experiments carried out over a wide variety of classification problems show that the genetic algorithm behaves better than, or at least, as well as the best of those heuristics and that subensembles with similar and sometimes better prediction accuracy can be obtained.

[1]  J. Ross Quinlan,et al.  Bagging, Boosting, and C4.5 , 1996, AAAI/IAAI, Vol. 1.

[2]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[3]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[4]  Gonzalo Martínez-Muñoz,et al.  Pruning in ordered bagging ensembles , 2006, ICML.

[5]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[6]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[7]  L. Breiman.  Arcing classifier (with discussion and a rejoinder by the author) , 1998 .

[8]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[9]  Christino Tamon,et al.  On the Boosting Pruning Problem , 2000, ECML.

[10]  G DietterichThomas.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees , 2000 .

[11]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[12]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[13]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[14]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[15]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[16]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .