Combining Diversity and Classification Accuracy for Ensemble Selection in Random Subspaces

An ensemble of classifiers has been shown to be effective in improving classifier performance. Two elements are believed to be viable in constructing an ensemble: a) the classification accuracy of each individual classifier; and b) diversity among the classifiers. Nevertheless, most works based on diversity suggest that there exists only weak correlation between diversity and ensemble accuracy. We propose compound diversity functions which combine the diversities with the classification accuracy of each individual classifier, and show that with Random subspaces ensemble creation method, there is a strong correlation between the proposed functions and ensemble accuracy. The statistical result indicates that compound diversity functions perform better than traditional diversity measures.

[1]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[2]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[3]  Naonori Ueda,et al.  Generalization error of ensemble estimators , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[4]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[5]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[6]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Dale Schuurmans,et al.  Boosting in the Limit: Maximizing the Margin of Learned Ensembles , 1998, AAAI/IAAI.

[8]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Pedro M. Domingos.  A Unifeid Bias-Variance Decomposition and its Applications , 2000, ICML.

[10]  Josef Kittler,et al.  Combining multiple classifiers by averaging or by multiplying? , 2000, Pattern Recognit..

[11]  Pedro M. Domingos.  A Unifeid Bias-Variance Decomposition and its Applications , 2000, ICML.

[12]  Fabio Roli,et al.  Dynamic classifier selection based on multiple classifier behaviour , 2001, Pattern Recognit..

[13]  Bogdan Gabrys,et al.  Analysis of the Correlation Between Majority Voting Error and the Diversity Measures in Multiple Classifier Systems , 2001 .

[14]  Fabio Roli,et al.  Design of effective neural network ensembles for image classification purposes , 2001, Image Vis. Comput..

[15]  Robert P. W. Duin,et al.  Bagging and Boosting for the Nearest Mean Classifier: Effects of Sample Size on Diversity and Accuracy , 2002, Multiple Classifier Systems.

[16]  Ludmila I. Kuncheva,et al.  Relationships between combination methods and measures of diversity in combining classifiers , 2002, Inf. Fusion.

[17]  Robert P. W. Duin,et al.  An experimental study on diversity for bagging and boosting with linear classifiers , 2002, Inf. Fusion.

[18]  Lawrence O. Hall,et al.  A New Ensemble Diversity Measure Applied to Thinning Ensembles , 2003, Multiple Classifier Systems.

[19]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[20]  Gareth James,et al.  Variance and Bias for General Loss Functions , 2003, Machine Learning.

[21]  Adel M. Alimi,et al.  Building Diverse Classifier Outputs to Evaluate the Behavior of Combination Methods: The Case of Two Classifiers , 2004, Multiple Classifier Systems.

[22]  Robert P. W. Duin,et al.  Combining Dissimilarity-Based One-Class Classifiers , 2004, Multiple Classifier Systems.

[23]  Robert Sabourin,et al.  Optimizing nearest neighbour in random subspaces using a multi-objective genetic algorithm , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[24]  Bogdan Gabrys,et al.  Classifier selection for majority voting , 2005, Inf. Fusion.

[25]  Gavin Brown,et al.  Between Two Extremes: Examining Decompositions of the Ensemble Objective Function , 2005, Multiple Classifier Systems.

[26]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[27]  Raymond J. Mooney,et al.  Creating diversity in ensembles using artificial data , 2005, Inf. Fusion.