Review of Classifier Combination Methods

Classifier combination methods have proved to be an effective tool to increase the performance of pattern recognition applications. In this chapter we review and categorize major advancements in this field. Despite a significant number of publications describing successful classifier combination implementations, the theoretical basis is still missing and achieved improvements are inconsistent. By introducing different categories of classifier combinations in this review we attempt to put forward more specific directions for future theoretical research. We also introduce a retraining effect and effects of locality based training as important properties of classifier combinations. Such effects have significant influence on the performance of combinations, and their study is necessary for complete theoretical understanding of combination algorithms.

[1]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Venu Govindaraju,et al.  Use of Lexicon Density in Evaluating Word Recognizers , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Josef Kittler,et al.  Experimental evaluation of expert fusion strategies , 1999, Pattern Recognit. Lett..

[4]  HoTin Kam.  The Random Subspace Method for Constructing Decision Forests , 1998 .

[5]  Venu Govindaraju,et al.  Classifier Combination Types for Biometric Applications , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[6]  Sargur N. Srihari,et al.  A theory of classifier combination: the neural network approach , 1995, Proceedings of 3rd International Conference on Document Analysis and Recognition.

[7]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[9]  E. Mandler,et al.  Combining the Classification Results of Independent Classifiers Based on the Dempster/Shafer Theory of Evidence , 1988 .

[10]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[11]  Venu Govindaraju,et al.  Architecture for Classifier Combination Using Entropy Measures , 2000, Multiple Classifier Systems.

[12]  Ludmila I. Kuncheva,et al.  Clustering-and-selection model for classifier combination , 2000, KES'2000. Fourth International Conference on Knowledge-Based Intelligent Engineering Systems and Allied Technologies. Proceedings (Cat. No.00TH8516).

[13]  Gerhard Rigoll,et al.  Combination of multiple classifiers for handwritten word recognition , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[14]  Roland Auckenthaler,et al.  Score Normalization for Text-Independent Speaker Verification Systems , 2000, Digit. Signal Process..

[15]  E. M. Kleinberg,et al.  Stochastic discrimination , 1990, Annals of Mathematics and Artificial Intelligence.

[16]  Thomas G. Dietterich,et al.  Error-Correcting Output Coding Corrects Bias and Variance , 1995, ICML.

[17]  Horst Bunke,et al.  Early feature stream integration versus decision level combination in a multiple classifier system for text line recognition , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[18]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[19]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[20]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[21]  David S. Doermann,et al.  Identifying script on word-level with informational confidence , 2005, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[22]  Stefan Jaeger.  Using informational confidence values for classifier combination: an experiment with combined on-line/off-line Japanese character recognition , 2004, Ninth International Workshop on Frontiers in Handwriting Recognition.

[23]  Peter J. Huber,et al.  Robust Statistics , 2005, Wiley Series in Probability and Statistics.

[24]  Shirley Dex,et al.  JR 旅客販売総合システム（マルス）における運用及び管理について , 1991 .

[25]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[26]  Arun Ross,et al.  Score normalization in multimodal biometric systems , 2005, Pattern Recognit..

[27]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[28]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[29]  Horst Bunke,et al.  New Boosting Algorithms for Classification Problems with Large Number of Classes Applied to a Handwritten Word Recognition Task , 2003, Multiple Classifier Systems.

[30]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[31]  Venu Govindaraju,et al.  Use of Lexicon Density in Evaluating Word Recognizers , 2000, Multiple Classifier Systems.

[32]  Robert Tibshirani,et al.  Bias, Variance and Prediction Error for Classification Rules , 1996 .

[33]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[34]  Julian Fiérrez,et al.  Bayesian adaptation for user-dependent multimodal biometric authentication , 2005, Pattern Recognit..

[35]  Fabio Roli,et al.  Analysis of error-reject trade-off in linearly combined multiple classifiers , 2004, Pattern Recognit..

[36]  S. Tulyakov,et al.  Identification Model for Classifier Combinations , 2006, 2006 Biometrics Symposium: Special Session on Research at the Biometric Consortium Conference.

[37]  Masaki Nakagawa,et al.  Accumulated-Recognition-Rate Normalization for Combining Multiple On/Off-Line Japanese Character Classifiers Tested on a Large Database , 2003, Multiple Classifier Systems.

[38]  Pat Langley,et al.  Editorial: On Machine Learning , 1986, Machine Learning.

[39]  Ching Y. Suen,et al.  A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Venu Govindaraju,et al.  Deriving Pseudo-Probabilities of Correctness Given Scores (DPPS) , 2003 .

[41]  Louis Vuurpijl,et al.  An overview and comparison of voting methods for pattern recognition , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[42]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[43]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[44]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[45]  Kevin W. Bowyer,et al.  Combination of Multiple Classifiers Using Local Accuracy Estimates , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Dario Maio,et al.  Combining Fingerprint Classifiers , 2000, Multiple Classifier Systems.

[47]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[48]  Paul D. Gader,et al.  Fusion of handwritten word classifiers , 1996, Pattern Recognit. Lett..

[49]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[50]  Arun Ross,et al.  Learning user-specific parameters in a multibiometric system , 2002, Proceedings. International Conference on Image Processing.

[51]  Jin Hyung Kim,et al.  A probabilistic framework for combining multiple classifiers at abstract level , 1997, Proceedings of the Fourth International Conference on Document Analysis and Recognition.

[52]  Kagan Tumer,et al.  Linear and Order Statistics Combiners for Pattern Classification , 1999, ArXiv.

[53]  Ch Chen,et al.  Pattern recognition and artificial intelligence , 1976 .

[54]  Eugene M. Kleinberg,et al.  On the Algorithmic Implementation of Stochastic Discrimination , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[55]  Masaki Nakagawa,et al.  A new warping technique for normalizing likelihood of multiple classifiers and its effectiveness in combined on-line/off-line japanese character recognition , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[56]  S. Jaeger.  Informational classifier fusion , 2004, ICPR 2004.

[57]  Fabio Roli,et al.  Performance Analysis and Comparison of Linear Combiners for Classifier Fusion , 2002, SSPR/SPR.

[58]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[59]  A. Sharkey.  Linear and Order Statistics Combiners for Pattern Classification , 1999 .

[60]  Aaron E. Rosenberg,et al.  Speaker background models for connected digit password speaker verification , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[61]  I KunchevaLudmila.  A Theoretical Study on Six Classifier Fusion Strategies , 2002 .

[62]  Fabio Roli,et al.  Dynamic classifier selection based on multiple classifier behaviour , 2001, Pattern Recognit..

[63]  Michael C. Fairhurst,et al.  Trainable Multiple Classifier Schemes for Handwritten Character Recognition , 2002, Multiple Classifier Systems.

[64]  John Law,et al.  Robust Statistics—The Approach Based on Influence Functions , 1986 .

[65]  J.P. Campbell,et al.  Allowing good impostors to test , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[66]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[67]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[68]  Ludmila I. Kuncheva,et al.  A Theoretical Study on Six Classifier Fusion Strategies , 2002, IEEE Trans. Pattern Anal. Mach. Intell..