When Efficient Model Averaging Out-Performs Boosting and Bagging

The Bayes optimal classifier (BOC) is an ensemble technique used extensively in the statistics literature. However, compared to other ensemble techniques such as bagging and boosting, BOC is less known and rarely used in data mining. This is partly due to BOC being perceived as being inefficient and because bagging and boosting consistently outperforms a single model, which raises the question: “Do we even need BOC in datamining?”. We show that the answer to this question is “yes” by illustrating several recent efficient model averaging approximations to BOC can significantly outperform bagging and boosting in realistic situations such as extensive class label noise, sample selection bias and many-class problems. That model averaging techniques outperform bagging and boosting in these situations has not been published in the machine learning, mining or statistical communities to our knowledge.

[1]  Geoffrey I. Webb.  Further Experimental Evidence against the Utility of Occam's Razor , 1996, J. Artif. Intell. Res..

[2]  Pedro M. Domingos.  Bayesian Averaging of Classifiers and the Overfitting Problem , 2000, ICML.

[3]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[4]  Philip S. Yu,et al.  An improved categorization of classifier's sensitivity on sample selection bias , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[5]  John Langford,et al.  Suboptimal Behavior of Bayes and MDL in Classification Under Misspecification , 2004, COLT.

[6]  Adrian E. Raftery,et al.  Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors , 1999 .

[7]  Bianca Zadrozny,et al.  Learning and evaluating classifiers under sample selection bias , 2004, ICML.

[8]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[9]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[10]  Pedro M. Domingos.  Why Does Bagging Work? A Bayesian Account and its Implications , 1997, KDD.

[11]  Wray L. Buntine,et al.  A theory of learning classification rules , 1990 .

[12]  Kai Ming Ting,et al.  Maximizing Tree Diversity by Building Complete-Random Decision Trees , 2005, PAKDD.

[13]  Wei Fan,et al.  On the Optimality of Probability Estimation by Random Decision Trees , 2004, AAAI.

[14]  Wray L. Buntine.  Learning Classification Rules Using Bayes , 1989, ML.

[15]  G DietterichThomas.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees , 2000 .

[16]  David J. Hand,et al.  On Pruning and Averaging Decision Trees , 1995, ICML.

[17]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[18]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[19]  Philip S. Yu,et al.  Is random model better? On its accuracy and efficiency , 2003, Third IEEE International Conference on Data Mining.

[20]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[21]  Ian Davidson,et al.  An Ensemble Technique for Stable Learners with Performance Bounds , 2004, AAAI.