A dynamic overproduce-and-choose strategy for the selection of classifier ensembles

The overproduce-and-choose strategy, which is divided into the overproduction and selection phases, has traditionally focused on finding the most accurate subset of classifiers at the selection phase, and using it to predict the class of all the samples in the test data set. It is therefore, a static classifier ensemble selection strategy. In this paper, we propose a dynamic overproduce-and-choose strategy which combines optimization and dynamic selection in a two-level selection phase to allow the selection of the most confident subset of classifiers to label each test sample individually. The optimization level is intended to generate a population of highly accurate candidate classifier ensembles, while the dynamic selection level applies measures of confidence to reveal the candidate ensemble with the highest degree of confidence in the current decision. Experimental results conducted to compare the proposed method to a static overproduce-and-choose strategy and a classical dynamic classifier selection approach demonstrate that our method outperforms both these selection-based methods, and is also more efficient in terms of performance than combining the decisions of all classifiers in the initial pool.

[1]  Gian Luca Marcialis,et al.  A study on the performances of dynamic classifier selection based on local accuracy estimation , 2005, Pattern Recognit..

[2]  Ludmila I. Kuncheva,et al.  Switching between selection and fusion in combining classifiers: an experiment , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[3]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[4]  Pierre Loonis,et al.  A multiple classifier system using ambiguity rejection for clustering-classification cooperation , 2000 .

[5]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[6]  Luiz Eduardo Soares de Oliveira,et al.  Automatic Recognition of Handwritten Numerical Strings: A Recognition and Verification Strategy , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Robert Sabourin,et al.  Classification system optimization with multi-objective genetic algorithms , 2006 .

[8]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Kevin W. Bowyer,et al.  Combination of multiple classifiers using local accuracy estimates , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Robert Sabourin,et al.  Single and Multi-Objective Genetic Algorithms for the Selection of Ensemble of Classifiers , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[11]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[12]  Robert Sabourin,et al.  Optimizing nearest neighbour in random subspaces using a multi-objective genetic algorithm , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[13]  Robert Sabourin,et al.  From dynamic classifier selection to dynamic ensemble selection , 2008, Pattern Recognit..

[14]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[15]  Jack Sklansky,et al.  Automated design of linear tree classifiers , 1990, Pattern Recognit..

[16]  Dale Schuurmans,et al.  Boosting in the Limit: Maximizing the Margin of Learned Ensembles , 1998, AAAI/IAAI.

[17]  Ludmila I. Kuncheva,et al.  Clustering-and-selection model for classifier combination , 2000, KES'2000. Fourth International Conference on Knowledge-Based Intelligent Engineering Systems and Allied Technologies. Proceedings (Cat. No.00TH8516).

[18]  Noel E. Sharkey,et al.  The "Test and Select" Approach to Ensemble Combination , 2000, Multiple Classifier Systems.

[19]  R. K. Ursem.  Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[20]  William B. Yates,et al.  Engineering Multiversion Neural-Net Systems , 1996, Neural Computation.

[21]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[22]  L. K. Hansen,et al.  The Error-Reject Tradeoff , 1997 .

[23]  Bogdan Gabrys,et al.  Classifier selection for majority voting , 2005, Inf. Fusion.

[24]  Xindong Wu,et al.  Dynamic classifier selection for effective mining from noisy data streams , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[25]  H.W. Shin,et al.  Selected tree classifier combination based on both accuracy and error diversity , 2005, Pattern Recognit..

[26]  So Young Sohn,et al.  Combining both ensemble and dynamic classifier selection schemes for prediction of mobile internet subscribers , 2003, Expert Syst. Appl..

[27]  Robert Sabourin,et al.  Ambiguity-guided dynamic selection of ensemble of classifiers , 2007, 2007 10th International Conference on Information Fusion.

[28]  Padraig Cunningham,et al.  Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error , 2001, ECML.

[29]  David W. Corne,et al.  No Free Lunch and Free Leftovers Theorems for Multiobjective Optimisation Problems , 2003, EMO.

[30]  Anne M. P. Canuto,et al.  Using Accuracy and Diversity to Select Classifiers to Build Ensembles , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[31]  Nojun Kwak,et al.  Feature extraction for classification problems and its application to face recognition , 2008, Pattern Recognit..

[32]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[33]  Patrick J. Grother,et al.  NIST Special Database 19 Handprinted Forms and Characters Database , 1995 .

[34]  Lorenzo Bruzzone,et al.  An experimental comparison of neural and statistical non-parametric algorithms for supervised classification of remote-sensing images , 1996, Pattern Recognit. Lett..

[35]  Fabio Roli,et al.  Dynamic classifier selection based on multiple classifier behaviour , 2001, Pattern Recognit..

[36]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Maneesha Singh,et al.  A dynamic classifier selection and combination approach to image region labelling , 2005, Signal Process. Image Commun..

[38]  Baozong Yuan,et al.  Multiple classifiers combination by clustering and selection , 2001, Inf. Fusion.

[39]  Anne M. P. Canuto,et al.  Applying Weights in the Functioning of the Dynamic Classifier Selection Method , 2006, 2006 Ninth Brazilian Symposium on Neural Networks (SBRN'06).

[40]  Pierre Valin,et al.  New initial basic probability assignments for multiple classifiers , 2002, SPIE Defense + Commercial Sensing.

[41]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..