A Survey of Decision Fusion and Feature Fusion Strategies for Pattern Classification

Abstract For any pattern classification task, an increase in data size, number of classes, dimension of the feature space, and interclass separability affect the performance of any classifier. A single classifier is generally unable to handle the wide variability and scalability of the data in any problem domain. Most modern techniques of pattern classification use a combination of classifiers and fuse the decisions provided by the same, often using only a selected set of appropriate features for the task. The problem of selection of a useful set of features and discarding the ones which do not provide class separability are addressed in feature selection and fusion tasks. This paper presents a review of the different techniques and algorithms used in decision fusion and feature fusion strategies, for the task of pattern classification. A survey of the prominent techniques used for decision fusion, feature selection, and fusion techniques has been discussed separately. The different techniques used for fusion have been categorized based on the applicability and methodology adopted for classification. A novel framework has been proposed by us, combining both the concepts of decision fusion and feature fusion to increase the performance of classification. Experiments have been done on three benchmark datasets to prove the robustness of combining feature fusion and decision fusion techniques.

[1]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[2]  Sukhendu Das,et al.  Dual space based face recognition using feature fusion , 2006 .

[3]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Paul C. Smits,et al.  Multiple classifier systems for supervised remote sensing image classification based on dynamic classifier selection , 2002, IEEE Trans. Geosci. Remote. Sens..

[5]  Sukhendu Das,et al.  Unsupervised texture segmentation using feature selection and fusion , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[6]  Qiang Yang,et al.  Feature selection in a kernel space , 2007, ICML '07.

[7]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[8]  Ching Y. Suen,et al.  Optimal combinations of pattern classifiers , 1995, Pattern Recognition Letters.

[9]  Kevin W. Bowyer,et al.  Combination of multiple classifiers using local accuracy estimates , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Jing Hua,et al.  Localized feature selection for clustering , 2008, Pattern Recognit. Lett..

[11]  Juan Domingo,et al.  Combining similarity measures in content-based image retrieval , 2008, Pattern Recognit. Lett..

[12]  Liangpei Zhang,et al.  Texture feature fusion for high resolution satellite image classification , 2005, International Conference on Computer Graphics, Imaging and Visualization (CGIV'05).

[13]  X. Guorong,et al.  Bhattacharyya distance feature selection , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[14]  Zhipeng Wang,et al.  Canonical Correlation Feature Selection for Sensors With Overlapping Bands: Theory and Application , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Fuad Rahman,et al.  A new multi-expert decision combination algorithm and its application to the detection of circumscribed masses in digital mammograms , 2001, Pattern Recognit..

[16]  Johannes R. Sveinsson,et al.  Multiple classifiers applied to multisource remote sensing data , 2002, IEEE Trans. Geosci. Remote. Sens..

[17]  Parameswaran Ramanathan,et al.  Distributed target classification and tracking in sensor networks , 2003 .

[18]  Yan Liu,et al.  A new method of feature fusion and its application in image recognition , 2005, Pattern Recognit..

[19]  Ching Y. Suen,et al.  A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Yanqing Zhang,et al.  Combining SVM Classifiers Using Genetic Fuzzy Systems Based on AUC for Gene Expression Data Analysis , 2007, ISBRA.

[21]  Karl-Michael Schneider,et al.  A New Feature Selection Score for Multinomial Naive Bayes Text Classification Based on KL-Divergence , 2004, ACL.

[22]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[23]  Belur V. Dasarathy,et al.  Decision fusion , 1994 .

[24]  Hong Yan,et al.  Human Face Image Recognition: An Evidence Aggregation Approach , 1998, Comput. Vis. Image Underst..

[25]  Huan Liu,et al.  Feature selection for clustering - a filter solution , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[26]  Huan Liu,et al.  Feature Selection for Classification , 1997, Intell. Data Anal..

[27]  Mineichi Kudo,et al.  Comparison of algorithms that select features for pattern classifiers , 2000, Pattern Recognit..

[28]  Carla E. Brodley,et al.  Feature Subset Selection and Order Identification for Unsupervised Learning , 2000, ICML.

[29]  Jian Yang,et al.  Generalized K-L transform based combined feature extraction , 2002, Pattern Recognit..

[30]  Belur V. Dasarathy,et al.  Nearest neighbor (NN) norms: NN pattern classification techniques , 1991 .

[31]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[32]  S. Manikandan,et al.  A Mathematical Approach for Feature Selection & Image Retrieval of Ultra Sound Kidney Image Databases , 2008 .

[33]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Jeffrey C. Schlimmer,et al.  Efficiently Inducing Determinations: A Complete and Systematic Search Algorithm that Uses Optimal Pruning , 1993, ICML.

[35]  Wen Gao,et al.  Mean-Shift Blob Tracking with Adaptive Feature Selection and Scale Adaptation , 2007, 2007 IEEE International Conference on Image Processing.

[36]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[37]  Sukhendu Das,et al.  A Fast Supervised Method of Feature Ranking and Selection for Pattern Classification , 2009, PReMI.

[38]  Domenec Puig,et al.  Automatic texture feature selection for image pixel classification , 2006, Pattern Recognit..

[39]  Ludmila I. Kuncheva,et al.  Switching between selection and fusion in combining classifiers: an experiment , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[40]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[41]  Huan Liu,et al.  Toward integrating feature selection algorithms for classification and clustering , 2005, IEEE Transactions on Knowledge and Data Engineering.

[42]  Pavel Pudil,et al.  Introduction to Statistical Pattern Recognition , 2006 .

[43]  Huan Liu,et al.  A Probabilistic Approach to Feature Selection - A Filter Solution , 1996, ICML.

[44]  Nicolaj Søndberg-Madsen,et al.  Unsupervised Feature Subset Selection , 2003 .

[45]  Mark A. Hall,et al.  Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning , 1999, ICML.

[46]  Manfred K. Warmuth,et al.  The Weighted Majority Algorithm , 1994, Inf. Comput..

[47]  Pheng-Ann Heng,et al.  Feature fusion method based on canonical correlation analysis and handwritten character recognition , 2004, ICARCV 2004 8th Control, Automation, Robotics and Vision Conference, 2004..

[48]  Paul D. Gader,et al.  Fusion of handwritten word classifiers , 1996, Pattern Recognit. Lett..

[49]  Robert I. Damper,et al.  A fast separability-based feature-selection method for high-dimensional remotely sensed image classification , 2008, Pattern Recognit..

[50]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[51]  Jeng-Shyang Pan,et al.  Discriminant Feature Fusion Strategy for Supervised Learning , 2006, 2006 International Conference on Intelligent Information Hiding and Multimedia.

[52]  C. A. Murthy,et al.  Unsupervised Feature Selection Using Feature Similarity , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  Jean-Philippe Thiran,et al.  Relevant Feature Selection for Audio-Visual Speech Recognition , 2007, 2007 IEEE 9th Workshop on Multimedia Signal Processing.

[54]  Huan Liu,et al.  Searching for Interacting Features , 2007, IJCAI.

[55]  Jian Yang,et al.  Feature fusion: parallel strategy vs. serial strategy , 2003, Pattern Recognit..

[56]  Björn Waske,et al.  Classifying Multilevel Imagery From SAR and Optical Sensors by Decision Fusion , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[57]  Naren Ramakrishnan,et al.  Algorithms for Feature Selection in Rank-Order Spaces , 2005 .

[58]  Zehra Cataltepe,et al.  A PCA/ICA based feature selection method and its application for corn fungi detection , 2007, 2007 15th European Signal Processing Conference.

[59]  Weiguo Gong,et al.  Feature Selection Based on KPCA, SVM and GSFS for Face Recognition , 2005, ICAPR.

[60]  Ioannis B. Theocharis,et al.  A multilayered neuro-fuzzy classifier with self-organizing properties , 2008, Fuzzy Sets Syst..

[61]  Raymond N. J. Veldhuis,et al.  Threshold-optimized decision-level fusion and its application to biometrics , 2009, Pattern Recognit..

[62]  Saurabh Prasad,et al.  Decision Fusion With Confidence-Based Weight Assignment for Hyperspectral Target Recognition , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[63]  Kohji Fukunaga,et al.  Introduction to Statistical Pattern Recognition-Second Edition , 1990 .

[64]  James C. Bezdek,et al.  Decision templates for multiple classifier fusion: an experimental comparison , 2001, Pattern Recognit..