Cluster-Oriented Ensemble Classifier: Impact of Multicluster Characterization on Ensemble Classifier Learning

This paper presents a novel cluster-oriented ensemble classifier. The proposed ensemble classifier is based on original concepts such as learning of cluster boundaries by the base classifiers and mapping of cluster confidences to class decision using a fusion classifier. The categorized data set is characterized into multiple clusters and fed to a number of distinctive base classifiers. The base classifiers learn cluster boundaries and produce cluster confidence vectors. A second level fusion classifier combines the cluster confidences and maps to class decisions. The proposed ensemble classifier modifies the learning domain for the base classifiers and facilitates efficient learning. The proposed approach is evaluated on benchmark data sets from UCI machine learning repository to identify the impact of multicluster boundaries on classifier learning and classification accuracy. The experimental results and two-tailed sign test demonstrate the superiority of the proposed cluster-oriented ensemble classifier over existing ensemble classifiers published in the literature.

[1]  Juan José Rodríguez Diez,et al.  Boosting recombined weak classifiers , 2008, Pattern Recognit. Lett..

[2]  Robi Polikar,et al.  An Ensemble-Based Incremental Learning Approach to Data Fusion , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[3]  Terry Windeatt,et al.  Accuracy/Diversity and Ensemble MLP Classifier Design , 2006, IEEE Transactions on Neural Networks.

[4]  Loris Nanni,et al.  FuzzyBagging: A novel ensemble of classifiers , 2006, Pattern Recognit..

[5]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[6]  Mohamed S. Kamel,et al.  A generalized adaptive ensemble generation and aggregation approach for multiple classifier systems , 2009, Pattern Recognit..

[7]  David Masip,et al.  Geometry-Based Ensembles: Toward a Structural Characterization of the Classification Boundary , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[9]  Jude W. Shavlik,et al.  Combining the Predictions of Multiple Classifiers: Using Competitive Learning to Initialize Neural Networks , 1995, IJCAI.

[10]  Ernest Valveny,et al.  Optimal Classifier Fusion in a Non-Bayesian Probabilistic Framework , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  R. Polikar,et al.  Ensemble based systems in decision making , 2006, IEEE Circuits and Systems Magazine.

[12]  Luiz Eduardo Soares de Oliveira,et al.  Pairwise fusion matrix for combining classifiers , 2007, Pattern Recognit..

[13]  Rich Caruana,et al.  An empirical comparison of supervised learning algorithms , 2006, ICML.

[14]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[15]  Mohamed S. Kamel,et al.  Adaptive fusion and co-operative training for classifier ensembles , 2006, Pattern Recognit..

[16]  Lawrence O. Hall,et al.  A New Ensemble Diversity Measure Applied to Thinning Ensembles , 2003, Multiple Classifier Systems.

[17]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[18]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[19]  James C. Bezdek,et al.  Decision templates for multiple classifier fusion: an experimental comparison , 2001, Pattern Recognit..

[20]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[21]  Maliha S. Nash,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 2001, Technometrics.

[22]  Fabio Roli,et al.  A Theoretical Analysis of Bagging as a Linear Combination of Classifiers , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[25]  Nicolás García-Pedrajas,et al.  Constructing Ensembles of Classifiers by Means of Weighted Instance Selection , 2009, IEEE Transactions on Neural Networks.

[26]  E. Forgy,et al.  Cluster analysis of multivariate data : efficiency versus interpretability of classifications , 1965 .

[27]  Robi Polikar,et al.  Learn$^{++}$ .NC: Combining Ensemble of Classifiers With Dynamically Weighted Consult-and-Vote for Efficient Incremental Learning of New Classes , 2009, IEEE Transactions on Neural Networks.

[28]  Daniel Hernández-Lobato,et al.  An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[30]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[31]  Hannu Oja,et al.  Classification Based on Hybridization of Parametric and Nonparametric Classifiers , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.