Using Clustering for Generating Diversity in Classifier Ensemble

In the past decade many new methods were proposed for creating diverse classifiers due to combination. In this paper a new method for constructing an ensemble is proposed which uses clustering technique to generate perturbation in training datasets. Main presumption of this method is that the clustering algorithm used can find the natural groups of data in feature space. During testing, the classifiers whose votes are considered as being reliable are combined using majority voting. This method of combination outperforms the ensemble of all classifiers considerably on several real and artificial datasets.

[1]  Lakhmi C. Jain,et al.  Designing classifier fusion systems by genetic algorithms , 2000, IEEE Trans. Evol. Comput..

[2]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[3]  Horst Bunke,et al.  Creation of classifier ensembles for handwritten word recognition using feature selection algorithms , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[4]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[5]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[6]  William F. Punch,et al.  Optimizing Classification Ensembles via a Genetic Algorithm for a Web-Based Educational System , 2004, SSPR/SPR.

[7]  H. Alizadeh,et al.  Divide & Conquer Classification and Optimization by Genetic Algorithm , 2008, 2008 Third International Conference on Convergence and Hybrid Information Technology.

[8]  Sandrine Dudoit,et al.  Bagging to Improve the Accuracy of A Clustering Procedure , 2003, Bioinform..

[9]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[10]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[11]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[12]  Oleksandr Makeyev,et al.  Neural network with ensembles , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[13]  Johannes Gehrke,et al.  Scalable Classification and Regression Tree Construction , 2003 .

[14]  Mahmood Fathy,et al.  Improved Face Detection Using Spatial Histogram Features , 2008, IPCV.

[15]  David G. Stork,et al.  Pattern Classification , 1973 .

[16]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[17]  Zoran Obradovic,et al.  Effective pruning of neural network classifier ensembles , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[18]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[19]  Fabio Roli,et al.  Methods for Designing Multiple Classifier Systems , 2001, Multiple Classifier Systems.

[20]  Pablo M. Granitto,et al.  Selecting diverse members of neural network ensembles , 2000, Proceedings. Vol.1. Sixth Brazilian Symposium on Neural Networks.

[21]  L. Shapley,et al.  Optimizing group judgmental accuracy in the presence of interdependencies , 1984 .

[22]  Thomas G. Dietterich.  Machine-Learning Research Four Current Directions , 1997 .

[23]  Jianxin Wu,et al.  Genetic Algorithm based Selective Neural Network Ensemble , 2001, IJCAI.

[24]  Fu Qiang.  PSO-based approach for neural network ensembles , 2004 .

[25]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[26]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[27]  Amanda J. C. Sharkey,et al.  Combining Artificial Neural Nets: Ensemble and Modular Multi-Net Systems , 1999 .

[28]  Bruce E. Rosen,et al.  Ensemble Learning Using Decorrelated Neural Networks , 1996, Connect. Sci..

[29]  W. Punch,et al.  Mining Feature Importance : Applying Evolutionary Algorithms within a Web-based Educational System , 2004 .

[30]  Mojtaba Vahidi-Asl,et al.  Learn to Detect Phishing Scams Using Learning and Ensemble ?Methods , 2007, 2007 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Workshops.

[31]  Raymond J. Mooney,et al.  Constructing Diverse Classifier Ensembles using Artificial Training Examples , 2003, IJCAI.

[32]  Louisa Lam,et al.  Classifier Combinations: Implementations and Theoretical Issues , 2000, Multiple Classifier Systems.

[33]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[34]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[35]  David W. Opitz,et al.  Actively Searching for an E(cid:11)ective Neural-Network Ensemble , 1996 .

[36]  Thomas G. Dietterich.  Machine-Learning Research , 1997, AI Mag..