Classifiers Combination Techniques: A Comprehensive Review

In critical applications, such as medical diagnosis, security related systems, and so on, the cost or risk of action taking based on incorrect classification can be very high. Hence, combining expert opinions before taking decision can substantially increase the reliability of such systems. Such pattern recognition systems base their final decision on evidence collected from different classifiers. Such evidence can be of data type, feature type, or classifier type. Common problems in pattern recognition, such as curse of dimensionality, and small sample data size, among others, have also prompted researchers into seeking new approaches for combining evidences. This paper presents a criteria-based framework for multi-classifiers combination techniques and their areas of applications. The criteria discussed here include levels of combination, types of thresholding, adaptiveness of the combination, and ensemble-based approaches. The strengths and weaknesses of each of these categories are discussed in details. Following this analysis, we provide our perspective on the outlook of this area of research and open problems. The lack of a well-formulated theoretical framework for analyzing the performance of combination techniques is shown to provide a fertile ground for further research. In addition to summarizing the existing work, this paper also updates and complements the latest developments in this area of research.

[1]  Saurabh Prasad,et al.  Decision-Level Fusion of Spectral Reflectance and Derivative Information for Robust Hyperspectral Land Cover Classification , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Salim Chitroub,et al.  Classifier combination and score level fusion: concepts and practical aspects , 2010 .

[3]  Diego H. Milone,et al.  Emotion Recognition in Never-Seen Languages Using a Novel Ensemble Method with Emotion Profiles , 2017, IEEE Transactions on Affective Computing.

[4]  Haibo He,et al.  SSC: A Classifier Combination Method Based on Signal Strength , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Sankhadip Saha,et al.  Multiple classifier combination technique for sensor drift compensation using ANN & KNN , 2014, 2014 IEEE International Advance Computing Conference (IACC).

[6]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[7]  Horst Bunke,et al.  A new combination scheme for HMM-based classifiers and its application to handwriting recognition , 2002, Object recognition supported by user interaction for service robots.

[8]  Chuang Liu,et al.  Multiple classifier combination for land cover classification of remote sensing image , 2010, The 2nd International Conference on Information Science and Engineering.

[9]  J. Bezdek,et al.  On combining multiple classifiers by fuzzy templates , 1998, 1998 Conference of the North American Fuzzy Information Processing Society - NAFIPS (Cat. No.98TH8353).

[10]  Mauro Barni,et al.  A Framework for Decision Fusion in Image Forensics Based on Dempster–Shafer Theory of Evidence , 2013, IEEE Transactions on Information Forensics and Security.

[11]  Ethem Alpaydin,et al.  Incremental construction of classifier and discriminant ensembles , 2009, Inf. Sci..

[12]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Jian Hou,et al.  A graph-theoretic approach to classifier combination , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Imran Naseem,et al.  Combining Classifiers Using the Dempster Shafer Theory of Evidence , 2010 .

[15]  Ludmila I. Kuncheva,et al.  A Theoretical Study on Six Classifier Fusion Strategies , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Palma Blonda,et al.  Neural network ensemble and support vector machine classifiers for the analysis of remotely sensed data: a comparison , 2002, IEEE International Geoscience and Remote Sensing Symposium.

[17]  Ethem Alpaydin,et al.  Eigenclassifiers for combining correlated classifiers , 2012, Inf. Sci..

[18]  Khaled Hassanein,et al.  A comparative study of combination schemes for an ensemble of digit recognition neural networks , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[19]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Gaurav Jain,et al.  An approach to text classification using dimensionality reduction and combination of classifiers , 2004, Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004..

[21]  Hong Yan,et al.  Nighttime Vehicle Detection Based on Bio-Inspired Image Enhancement and Weighted Score-Level Feature Fusion , 2017, IEEE Transactions on Intelligent Transportation Systems.

[22]  Juan Hu,et al.  Offline Signature Verification Using Real Adaboost Classifier Combination of Pseudo-dynamic Features , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[23]  Thierry Denoeux,et al.  Pairwise classifier combination using belief functions , 2007, Pattern Recognit. Lett..

[24]  Josef Kittler,et al.  Combining classifiers: A theoretical framework , 1998, Pattern Analysis and Applications.

[25]  Maurice Kettner,et al.  Spark plug fault recognition based on sensor fusion and classifier combination using Dempster–Shafer evidence theory , 2015 .

[26]  Lei Gao,et al.  A New Model of Combining Multiple Classifiers Based on Neural Network , 2013, 2013 Fourth International Conference on Emerging Intelligent Data and Web Technologies.

[27]  Jon Atli Benediktsson,et al.  Decision Fusion for the Classification of Urban Remote Sensing Images , 2006, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Yi Yang,et al.  Combination of heterogeneous multiple classifiers based on evidence theory , 2007, 2007 International Conference on Wavelet Analysis and Pattern Recognition.

[29]  Claudio De Stefano,et al.  A Novel Naive Bayes Voting Strategy for Combining Classifiers , 2012, 2012 International Conference on Frontiers in Handwriting Recognition.

[30]  J. R. Parker,et al.  Combining multiple non-homogeneous classifiers: an empirical approach , 2002, Proceedings First IEEE International Conference on Cognitive Informatics.

[31]  Rizoan Toufiq,et al.  Face recognition system using soft-output classifier fusion method , 2016, 2016 2nd International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE).

[32]  Dilek Z. Hakkani-Tür,et al.  Entropy Based Classifier Combination for Sentence Segmentation , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[33]  Alicja R. Rudnicka,et al.  Microaneurysm detection in retinal images using an ensemble classifier , 2016, 2016 Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA).

[34]  Andrey V. Bogdanov.  Neuroinspired Architecture for Robust Classifier Fusion of Multisensor Imagery , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[35]  Holger Schwenk,et al.  Using boosting to improve a hybrid HMM/neural network speech recognizer , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[36]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  T. D. Pham.  Combination of multiple classifiers using adaptive fuzzy integral , 2002, Proceedings 2002 IEEE International Conference on Artificial Intelligence Systems (ICAIS 2002).

[38]  Robert P. W. Duin,et al.  Experiments with Classifier Combining Rules , 2000, Multiple Classifier Systems.

[39]  Gerhard Rigoll,et al.  Combination of multiple classifiers for handwritten word recognition , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[40]  Fatos T. Yarman-Vural,et al.  Building Detection With Decision Fusion , 2013, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[41]  Fabio Roli,et al.  A theoretical and experimental analysis of linear combiners for multiple classifier systems , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Ponciano Jorge Escamilla-Ambrosio,et al.  Feature selection and ensemble of classifiers for Android malware detection , 2016, 2016 8th IEEE Latin-American Conference on Communications (LATINCOM).

[43]  Jian-Huang Lai,et al.  Linear Dependency Modeling for Classifier Fusion and Feature Combination , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Anupam Agrawal,et al.  Fusion Based Emotion Recognition System , 2016, 2016 International Conference on Computational Science and Computational Intelligence (CSCI).

[45]  Fuad Rahman,et al.  Decision combination of multiple classifiers for pattern classification: hybridisation of majority voting and divide and conquer techniques , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[46]  B. Lazzerini,et al.  A TSK fuzzy model for combining outputs of multiple classifiers , 2004, IEEE Annual Meeting of the Fuzzy Information, 2004. Processing NAFIPS '04..

[47]  Ofer Melnik,et al.  Mixed group ranks: preference and confidence in classifier combination , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Venu Govindaraju,et al.  Review of Classifier Combination Methods , 2008, Machine Learning in Document Analysis and Recognition.

[49]  Josef Kittler,et al.  A Framework for Classifier Fusion: Is It Still Needed? , 2000, SSPR/SPR.

[50]  Xia Xu,et al.  Hierarchical Guidance Filtering-Based Ensemble Classification for Hyperspectral Images , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[51]  Naoufel Werghi,et al.  Enhancing automatic polyp detection accuracy using fusion techniques , 2016, 2016 IEEE 59th International Midwest Symposium on Circuits and Systems (MWSCAS).

[52]  Ching Y. Suen,et al.  Optimal combinations of pattern classifiers , 1995, Pattern Recognit. Lett..

[53]  Mohamed A. Deriche,et al.  Arabic sign language recognition using the leap motion controller , 2014, 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE).

[54]  Ludmila I. Kuncheva,et al.  Switching between selection and fusion in combining classifiers: an experiment , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[55]  Cinthia Obladen de Almendra Freitas,et al.  Handwritten recognition with multiple classifiers for restricted lexicon , 2004, Proceedings. 17th Brazilian Symposium on Computer Graphics and Image Processing.

[56]  Sebastiano Impedovo,et al.  Knowledge-based methods for classifier combination: an experimental investigation , 1999, Proceedings 10th International Conference on Image Analysis and Processing.

[57]  Josef Kittler,et al.  A Unified Framework for Biometric Expert Fusion Incorporating Quality Measures , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Mohammed Abdul Azeem Siddiqui.  Fusion of ECG/EEG for improved automatic seizure detection using Dempster Shafer theory of evidence , 2011 .

[59]  Michael C. Fairhurst,et al.  Optimisation of multiple classifier systems using genetic algorithms , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[60]  Robi Polikar,et al.  Learn$^{++}$ .NC: Combining Ensemble of Classifiers With Dynamically Weighted Consult-and-Vote for Efficient Incremental Learning of New Classes , 2009, IEEE Transactions on Neural Networks.

[61]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[62]  M. Deriche,et al.  Arabic sign language recognition by decisions fusion using Dempster-Shafer theory of evidence , 2013, 2013 Computing, Communications and IT Applications Conference (ComComAp).

[63]  R. Polikar,et al.  Ensemble based systems in decision making , 2006, IEEE Circuits and Systems Magazine.

[64]  Eric Lecolinet,et al.  A multi-classifier combination strategy for the recognition of handwritten cursive words , 1993, Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR '93).

[65]  David J. Miller,et al.  Ensemble classification by critic-driven combining , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[66]  Z. Cataltepe,et al.  Co-training with adaptive Bayesian classifier combination , 2008, 2008 23rd International Symposium on Computer and Information Sciences.

[67]  Robert P. W. Duin,et al.  The combining classifier: to train or not to train? , 2002, Object recognition supported by user interaction for service robots.

[68]  Christian Igel,et al.  Active learning with support vector machines , 2014, WIREs Data Mining Knowl. Discov..

[69]  Sankhadip Saha,et al.  A novel approach to classify edible oil using multiple classifier fusion based on spectral data , 2016, 2016 International Conference on Intelligent Control Power and Instrumentation (ICICPI).

[70]  Rui Zhao,et al.  A boundary based classifier combination method , 2009, 2009 Chinese Control and Decision Conference.

[71]  Ho-Sub Yoon,et al.  Automated Speaker Recognition for Home Service Robots Using Genetic Algorithm and Dempster–Shafer Fusion Technique , 2009, IEEE Transactions on Instrumentation and Measurement.

[72]  Masaki Nakagawa,et al.  A new warping technique for normalizing likelihood of multiple classifiers and its effectiveness in combined on-line/off-line japanese character recognition , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[73]  Joachim Denzler,et al.  Adaptive Performance-Based Classifier Combination for Generic Object Recognition , 2005 .

[74]  Changshui Zhang,et al.  Classifier combination based on active learning , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[75]  Zehra Cataltepe,et al.  Classifier combination with kernelized eigenclassifiers , 2013, Proceedings of the 16th International Conference on Information Fusion.

[76]  Masaki Nakagawa,et al.  Precise Candidate Selection for Large Character Set Recognition by Confidence Evaluation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..