‘ Fuzzy ’ vs ‘ Non-fuzzy ’ in Combining Classifiers Designed by Boosting

Boosting is recognized as one of the most successful techniques for generating classifier ensembles. Typically, the classifier outputs are combined by the weighted majority vote. The purpose of this study is to demonstrate the advantages of some fuzzy combination methods for ensembles of classifiers designed by Boosting. We ran 2-fold cross-validation experiments on 6 benchmark data sets to compare the fuzzy and non-fuzzy combination methods. On the “fuzzy side” we used the fuzzy integral and the decision templates with different similarity measures. On the “non-fuzzy side” we tried the weighted majority vote as well as simple combiners such as the majority vote, minimum, maximum, average, product, and the Naive Bayes combination. In our experiments, the fuzzy combination methods performed consistently better than the nonfuzzy methods. The weighted majority vote showed a stable performance, though slightly inferior to the performance of the fuzzy combiners.

[1]  W. H. Pierce,et al.  IMPROVING RELIABILITY OF DIGITAL SYSTEMS BY REDUNDANCY AND ADAPTION , 1961 .

[2]  R. Yager.  On a general class of fuzzy connectives , 1980 .

[3]  Didier Dubois,et al.  Fuzzy sets and systems ' . Theory and applications , 2007 .

[4]  L. Shapley,et al.  Optimizing group judgmental accuracy in the presence of interdependencies , 1984 .

[5]  Didier Dubois,et al.  A review of fuzzy set aggregation connectives , 1985, Inf. Sci..

[6]  M. Sugeno,et al.  Multi-attribute classification using fuzzy integral , 1992, [1992 Proceedings] IEEE International Conference on Fuzzy Systems.

[7]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[8]  Roberto Battiti,et al.  Democracy in neural nets: Voting schemes for classification , 1994, Neural Networks.

[9]  Ching Y. Suen,et al.  A theoretical analysis of the application of majority voting to pattern recognition , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 3 - Conference C: Signal Processing (Cat. No.94CH3440-5).

[10]  Michel Grabisch,et al.  On equivalence classes of fuzzy connectives-the case of fuzzy integrals , 1995, IEEE Trans. Fuzzy Syst..

[11]  Ching Y. Suen,et al.  A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Sung-Bae Cho,et al.  Combining multiple neural networks by fuzzy integral for robust classification , 1995, IEEE Trans. Syst. Man Cybern..

[13]  Sung-Bae Cho,et al.  Multiple network fusion using fuzzy logic , 1995, IEEE Trans. Neural Networks.

[14]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[15]  Bernadette Bouchon-Meunier,et al.  Towards general measures of comparison of objects , 1996, Fuzzy Sets Syst..

[16]  Kagan Tumer,et al.  Analysis of decision boundaries in linearly combined neural classifiers , 1996, Pattern Recognit..

[17]  Kevin W. Bowyer,et al.  Combination of multiple classifiers using local accuracy estimates , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Isabelle Bloch.  Information combination operators for data fusion: a comparative review with classification , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[19]  Paul D. Gader,et al.  Fusion of handwritten word classifiers , 1996, Pattern Recognit. Lett..

[20]  Kagan Tumer,et al.  Error Correlation and Error Reduction in Ensemble Classifiers , 1996, Connect. Sci..

[21]  J. R. Sveinsson,et al.  Multistage classifiers optimized by neural networks and genetic algorithms , 1997 .

[22]  Didier Dubois,et al.  The three semantics of fuzzy sets , 1997, Fuzzy Sets Syst..

[23]  Ching Y. Suen,et al.  Application of majority voting to pattern recognition: an analysis of its behavior and performance , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[24]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[25]  D Wang,et al.  Use of fuzzy-logic-inspired features to improve bacterial recognition through classifier fusion , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[26]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Antanas Verikas,et al.  Soft combination of neural classifiers: A comparative study , 1999, Pattern Recognit. Lett..

[28]  Robert E. Schapire,et al.  Theoretical Views of Boosting , 1999, EuroCOLT.

[29]  Kagan Tumer,et al.  Linear and Order Statistics Combiners for Pattern Classification , 1999, ArXiv.

[30]  Thomas G. Dietterich.  Ensemble Methods in Machine Learning , 2000, Multiple Classifier Systems.

[31]  James C. Bezdek,et al.  Decision templates for multiple classifier fusion: an experimental comparison , 2001, Pattern Recognit..

[32]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[33]  L. Kuncheva,et al.  Combining classifiers: Soft computing solutions. , 2001 .

[34]  Fabio Roli,et al.  Performance Analysis and Comparison of Linear Combiners for Classifier Fusion , 2002, SSPR/SPR.

[35]  Luc Vandendorpe,et al.  Decision Level Fusion of Intramodal Personal Identity Verification Experts , 2002, Multiple Classifier Systems.

[36]  Robert E. Schapire,et al.  The Boosting Approach to Machine Learning An Overview , 2003 .

[37]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[38]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.