The impact of diversity on the accuracy of evidential classifier ensembles

Diversity being inherent in classifiers is widely acknowledged as an important issue in constructing successful classifier ensembles. Although many statistics have been employed in measuring diversity among classifiers to ascertain whether it correlates with ensemble performance in the literature, most of these measures are incorporated and explained in a non-evidential context. In this paper, we provide a modelling for formulating classifier outputs as triplet mass functions and a uniform notation for defining diversity measures. We then assess the relationship between diversity obtained by four pairwise and non-pairwise diversity measures and the improvement in accuracy of classifiers combined in different orders by Demspter's rule of combination, Smets' conjunctive rule, the Proportion and Yager's rules in the framework of belief functions. Our experimental results demonstrate that the accuracy of classifiers combined by Dempster's rule is not strongly correlated with the diversity obtained by the four measures, and the correlation between the diversity and the ensemble accuracy made by Proportion and Yager's rules is negative, which is not in favor of the claim that increasing diversity could lead to reduction of generalization error of classifier ensembles.

[1]  Shengli Wu,et al.  Measuring Impact of Diversity of Classifiers on the Accuracy of Evidential Ensemble Classifiers , 2010, IPMU.

[2]  Philippe Smets,et al.  The Combination of Evidence in the Transferable Belief Model , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Raymond J. Mooney,et al.  Creating diversity in ensembles using artificial data , 2005, Inf. Fusion.

[4]  Susan T. Dumais,et al.  The Combination of Text Classifiers Using Reliability Indicators , 2016, Information Retrieval.

[5]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Juha Röning,et al.  Methods for person identification on a pressure-sensitive floor: Experiments with multiple classifiers and reject option , 2008, Inf. Fusion.

[7]  T. Denœux.  Conjunctive and disjunctive combination of belief functions induced by nondistinct bodies of evidence , 2008 .

[8]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[9]  Eric Lefevre,et al.  Belief function combination and conflict management , 2002, Inf. Fusion.

[10]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[11]  Johannes R. Sveinsson,et al.  Random Forests for land cover classification , 2006, Pattern Recognit. Lett..

[12]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[13]  Gavin Brown,et al.  "Good" and "Bad" Diversity in Majority Vote Ensembles , 2010, MCS.

[14]  Ian Witten,et al.  Data Mining , 2000 .

[15]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[16]  Lian-zeng Zhang,et al.  Representation, independence, and combination of evidence in the Dempster-Shafer theory , 1994 .

[17]  Edward H. Shortliffe,et al.  A Method for Managing Evidential Reasoning in a Hierarchical Hypothesis Space , 1985, Artif. Intell..

[18]  R. Yager.  On the dempster-shafer framework and new combination rules , 1987, Inf. Sci..

[19]  Yaxin Bi,et al.  The combination of multiple classifiers using an evidential reasoning approach , 2008, Artif. Intell..

[20]  G. Yule.  On the Association of Attributes in Statistics: With Illustrations from the Material of the Childhood Society, &c , 1900 .

[21]  Weiru Liu,et al.  Analyzing the degree of conflict among belief functions , 2006, Artif. Intell..

[22]  Paul-André Monney,et al.  A belief function classifier based on information provided by noisy and dependent features , 2011, Int. J. Approx. Reason..

[23]  Thierry Denoeux,et al.  Classifier fusion in the Dempster-Shafer framework using optimized t-norm based combination rules , 2011, Int. J. Approx. Reason..

[24]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[25]  Ronald R. Yager,et al.  A knowledge‐based approach to adversarial decision making , 2008, Int. J. Intell. Syst..

[26]  G. Yule,et al.  On the association of attributes in statistics, with examples from the material of the childhood society, &c , 1900, Proceedings of the Royal Society of London.

[27]  Xin Yao,et al.  An analysis of diversity measures , 2006, Machine Learning.

[28]  Jon Atli Benediktsson,et al.  Decision Fusion for the Classification of Urban Remote Sensing Images , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[29]  Piero P. Bonissone,et al.  A fuzzy random forest , 2010, Int. J. Approx. Reason..

[30]  David A. Bell,et al.  EDM: A General Framework for Data Mining Based on Evidence Theory , 1996, Data Knowl. Eng..

[31]  Kagan Tumer,et al.  Linear and Order Statistics Combiners for Pattern Classification , 1999, ArXiv.

[32]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[33]  Yaxin Bi.  An efficient triplet-based algorithm for evidential reasoning , 2008 .

[34]  J. Fleiss,et al.  The Reliability of Dichotomous Judgments: Unequal Numbers of Judges per Subject , 1979 .

[35]  Yaxin Bi,et al.  COMBINING MULTIPLE CLASSIFIERS USING DEMPSTER'S RULE FOR TEXT CATEGORIZATION , 2007, Appl. Artif. Intell..

[36]  Zhi-Hua Zhou,et al.  Multi-information Ensemble Diversity , 2010, MCS.

[37]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[38]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[39]  Didier Dubois,et al.  On the unicity of dempster rule of combination , 1986, Int. J. Intell. Syst..

[40]  Philippe Smets,et al.  Analyzing the combination of conflicting belief functions , 2007, Inf. Fusion.