A Feature-Based Serial Approach to Classifier Combination

Abstract: A new approach to the serial multi-stage combination of classifiers is proposed. Each classifier in the sequence uses a smaller subset of features than the subsequent classifier. The classification provided by a classifier is rejected only if its decision is below a predefined confidence level. The approach is tested on a two-stage combination of k-Nearest Neighbour classifiers. The features to be used by the first classifier in the combination are selected by two stand-alone algorithms (Relief and Info-Fuzzy Network, or IFN) and a hybrid method, called ‘IFN + Relief’. The feature-based approach is shown empirically to provide a substantial decrease in the computational complexity, while maintaining the accuracy level of a single-stage classifier or even improving it.

[1]  Fabio Roli,et al.  A theoretical framework for dynamic classifier selection , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[2]  Ching Y. Suen,et al.  Combination of multiple classifier decisions for optical character recognition , 1997 .

[3]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Jon Atli Benediktsson,et al.  Multiple Classifier Systems , 2015, Lecture Notes in Computer Science.

[5]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[6]  Josef Kittler,et al.  Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Italy, June 21-23, 2000 Proceedings , 2000 .

[7]  Oded Maimon.  Knowledge Discovery and Data Mining : The Info-Fuzzy Network (IFN) Methodology , 2000 .

[8]  Hiroshi Motoda,et al.  Feature Selection for Knowledge Discovery and Data Mining , 1998, The Springer International Series in Engineering and Computer Science.

[9]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[11]  Hong Yan,et al.  Prototype optimization for nearest neighbor classifiers using a two-layer perceptron , 1993, Pattern Recognit..

[12]  G. Dunteman.  Principal Components Analysis , 1989 .

[13]  Josef Kittler,et al.  Floating search methods in feature selection , 1994, Pattern Recognit. Lett..

[14]  Fuad Rahman,et al.  Serial Combination of Multiple Experts: A Unified Evaluation , 1999, Pattern Analysis & Applications.

[15]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[17]  Jae-On Kim,et al.  Factor Analysis: Statistical Methods and Practical Issues , 1978 .

[18]  Ron Kohavi,et al.  Irrelevant Features and the Subset Selection Problem , 1994, ICML.

[19]  Keki B. Irani,et al.  Multi-interval discretization of continuos attributes as pre-processing for classi cation learning , 1993, IJCAI 1993.

[20]  Fabio Roli,et al.  Dynamic Classifier Selection , 2000, Multiple Classifier Systems.

[21]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[22]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[23]  Essaid Bouktache,et al.  A Fast Algorithm for the Nearest-Neighbor Classifier , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Robert E. Schapire,et al.  A Brief Introduction to Boosting , 1999, IJCAI.

[25]  Abraham Kandel,et al.  Information-theoretic algorithm for feature selection , 2001, Pattern Recognit. Lett..

[26]  Andrew R. Webb,et al.  Statistical Pattern Recognition , 1999 .

[27]  Jürgen Schürmann,et al.  Pattern classification , 2008 .

[28]  W. Eric L. Grimson,et al.  Constructing optimized prototypes for nearest neighbor classifiers , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[29]  Jari Kangas.  Comparison between two prototype representation schemes for a nearest neighbor classifier , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[30]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.

[31]  Robert P. W. Duin,et al.  Experiments with Classifier Combining Rules , 2000, Multiple Classifier Systems.

[32]  Larry A. Rendell,et al.  The Feature Selection Problem: Traditional Methods and a New Algorithm , 1992, AAAI.

[33]  Majid Ahmadi,et al.  Pattern classification using an efficient KNNR , 1992, Pattern Recognit..

[34]  Josef Kittler,et al.  Multiple Classifier Systems , 2004, Lecture Notes in Computer Science.

[35]  Horst Bunke,et al.  Handbook of Character Recognition and Document Image Analysis , 1997 .

[36]  James C. Bezdek,et al.  Multiple-prototype classifier design , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[37]  Pavel Paclík,et al.  Adaptive floating search methods in feature selection , 1999, Pattern Recognit. Lett..