Examining the Relationship Between Majority Vote Accuracy and Diversity in Bagging and Boosting

Much current research is undertaken into combining classiers to increase the classication accuracy. We show, by means of an enumerative example, how combining classiers can lead to much greater or lesser accuracy than each individual classier. Measures of diversity among the classiers taken from the literature are shown to only exhibit a weak relationship with majority vote accuracy. Two commonly used methods of designing classier ensembles, Bagging and Boosting, are examined on benchmark datasets. Bagging is shown to produce teams with little diversity or improvement in accuracy, while Boosting tends to produce more diverse classier teams showing an improvement in accuracy.

[1]  P. Sneath,et al.  Numerical Taxonomy , 1962, Nature.

[2]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[3]  Ching Y. Suen,et al.  Application of majority voting to pattern recognition: an analysis of its behavior and performance , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[4]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[5]  Bev Littlewood,et al.  Conceptual Modeling of Coincident Failures in Multiversion Software , 1989, IEEE Trans. Software Eng..

[6]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[7]  Derek Partridge,et al.  Software Diversity: Practical Statistics for Its Measurement and Exploitation | Draft Currently under Revision , 1996 .

[8]  Robert P. W. Duin,et al.  An experimental study on diversity for bagging and boosting with linear classifiers , 2002, Inf. Fusion.

[9]  G. Yule,et al.  On the association of attributes in statistics, with examples from the material of the childhood society, &c , 1900, Proceedings of the Royal Society of London.

[10]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[11]  Louisa Lam,et al.  Classifier Combinations: Implementations and Theoretical Issues , 2000, Multiple Classifier Systems.

[12]  David B. Skalak,et al.  The Sources of Increased Accuracy for Two Proposed Boosting Algorithms , 1996, AAAI 1996.

[13]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[14]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[15]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  David G. Stork,et al.  Pattern Classification , 1973 .

[17]  C. J. Whitaker,et al.  Ten measures of diversity in classifier ensembles: limits for two classifiers , 2001 .

[18]  Fabio Roli,et al.  Design of effective neural network ensembles for image classification purposes , 2001, Image Vis. Comput..

[19]  Robert P. W. Duin,et al.  Limits on the majority vote accuracy in classifier fusion , 2003, Pattern Analysis & Applications.

[20]  L. Breiman.  Arcing Classifiers , 1998 .

[21]  W. Conover.  Statistical Methods for Rates and Proportions , 1974 .

[22]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Yoav Freund,et al.  Discussion of the paper "Arcing Classifiers" by Leo Breiman , 1998 .

[24]  Ching Y. Suen,et al.  Optimal combinations of pattern classifiers , 1995, Pattern Recognit. Lett..