Two Bagging Algorithms with Coupled Learners to Encourage Diversity

In this paper, we present two ensemble learning algorithms which make use of boostrapping and out-of-bag estimation in an attempt to inherit the robustness of bagging to overfitting. As against bagging, with these algorithms learners have visibility on the other learners and cooperate to get diversity, a characteristic that has proved to be an issue of major concern to ensemble models. Experiments are provided using two regression problems obtained from UCI.

[1]  Kurt Hornik,et al.  Artificial Neural Networks — ICANN 2001 , 2001, Lecture Notes in Computer Science.

[2]  Leo Breiman,et al.  Using Iterated Bagging to Debias Regressions , 2001, Machine Learning.

[3]  Marc Toussaint,et al.  Extracting Motion Primitives from Natural Handwriting Data , 2006, ICANN.

[4]  Gavin Brown,et al.  Diversity in neural network ensembles , 2004 .

[5]  J. Friedman.  Stochastic gradient boosting , 2002 .

[6]  Alexander J. Smola,et al.  Support Vector Regression Machines , 1996, NIPS.

[7]  Raymond J. Mooney,et al.  Combining Bias and Variance Reduction Techniques for Regression Trees , 2005, ECML.

[8]  Massimiliano Pontil,et al.  Stability of Randomized Learning Algorithms , 2005, J. Mach. Learn. Res..

[9]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[10]  L. Breiman.  OUT-OF-BAG ESTIMATION , 1996 .

[11]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[12]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[13]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[14]  T. Poggio,et al.  Bagging Regularizes , 2002 .

[15]  Yves Grandvalet,et al.  Bagging Equalizes Influence , 2004, Machine Learning.

[16]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[17]  Claudio Moraga,et al.  Ensemble Learning with Local Diversity , 2006, ICANN.

[18]  Yves Grandvalet.  Bagging Can Stabilize without Reducing Variance , 2001, ICANN.