Dynamic fusion method using Localized Generalization Error Model

Multiple Classifier Systems (MCSs), which combine the outputs of a set of base classifiers, were proposed as a method to develop a more accurate classification system. One fundamental issue is how to combine the base classifiers. In this paper, a new dynamic fusion method named Localized Generalization Error Model Fusion Method (LFM) for MCSs is proposed. The Localized Generalization Error Model (L-GEM) has been used to estimate the local competence of base classifiers in MCSs. L-GEM provides a generalization error bound for unseen samples located within neighborhoods of testing samples. Base classifiers with lower generalization error bounds are assigned higher weights. In contrast to the current dynamic fusion methods, LFM estimates the local competence of base classifiers not only using the information of training error but also the sensitivity of classifier outputs. The additional effect of the sensitivity on the performance of model and the time complexity of the LFM are discussed and analyzed. Experimental results show that the MCSs using the LFM as a combination method outperform those using the other 21 dynamic fusion methods in terms of testing accuracy and time.

[1]  Marek Kurzynski,et al.  Application of Combining Classifiers Using Dynamic Weights to the Protein Secondary Structure Prediction - Comparative Analysis of Fusion Methods , 2006, ISBMDA.

[2]  Patrick P. K. Chan,et al.  A novel dynamic fusion method using localized generalization error model , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[3]  Francisco Herrera,et al.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power , 2010, Inf. Sci..

[4]  Mykola Pechenizkiy,et al.  Dynamic Integration with Random Forests , 2006, ECML.

[5]  Alexey Tsymbal,et al.  A Dynamic Integration Algorithm for an Ensemble of Classifiers , 1999, ISMIS.

[6]  Ludmila I. Kuncheva,et al.  Switching between selection and fusion in combining classifiers: an experiment , 2002, IEEE Trans. Syst. Man Cybern. Part B.

[7]  Xi-Zhao Wang,et al.  Improving Generalization of Fuzzy IF--THEN Rules by Maximizing Fuzzy Entropy , 2009, IEEE Transactions on Fuzzy Systems.

[8]  Wing W. Y. Ng,et al.  A multiple intelligent agent system for credit risk prediction via an optimization of localized generalization error with diversity , 2007 .

[9]  Anne M. P. Canuto,et al.  Applying Static and Dynamic Weight Measures in Ensemble Systems , 2008, 2008 10th Brazilian Symposium on Neural Networks.

[10]  Kevin Warwick,et al.  Robust initialisation of Gaussian radial basis function networks using partitioned k-means clustering , 1996 .

[11]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[12]  Xizhao Wang,et al.  Maximum Ambiguity-Based Sample Selection in Fuzzy Decision Tree Induction , 2012, IEEE Transactions on Knowledge and Data Engineering.

[13]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[14]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[15]  Robert Sabourin,et al.  From dynamic classifier selection to dynamic ensemble selection , 2008, Pattern Recognit..

[16]  Robi Polikar,et al.  Ensemble of classifiers based incremental learning with dynamic voting weight update , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[17]  Daniel S. Yeung,et al.  Localized Generalization Error Model and Its Application to Architecture Selection for Radial Basis Function Neural Network , 2007, IEEE Transactions on Neural Networks.

[18]  Patrick P. K. Chan,et al.  Active learning using localized generalization error of candidate sample as criterion , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[19]  Kevin W. Bowyer,et al.  Combination of Multiple Classifiers Using Local Accuracy Estimates , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Patrick P. K. Chan,et al.  Radial Basis Function network learning using localized generalization error bound , 2009, Inf. Sci..

[21]  Mohamad T. Musavi,et al.  On the training of radial basis function classifiers , 1992, Neural Networks.

[22]  Man Wai Mak,et al.  Genetic evolution of radial basis function centers for pattern classification , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[23]  Daniel S. Yeung,et al.  Multiple Classifier System with Feature Grouping for Intrusion Detection: Mutual Information Approach , 2005, KES.

[24]  Alexey Tsymbal,et al.  Bagging and Boosting with Dynamic Integration of Classifiers , 2000, PKDD.

[25]  Chuanyi Ji,et al.  Combinations of Weak Classifiers , 1996, NIPS.

[26]  Jaepil Ko,et al.  Dynamic Classifier Integration Method , 2005, Multiple Classifier Systems.

[27]  Christopher J. Merz,et al.  Combining Classifiers Using Correspondence Analysis , 1997, NIPS.

[28]  Kagan Tumer,et al.  Error Correlation and Error Reduction in Ensemble Classifiers , 1996, Connect. Sci..

[29]  Ajith Abraham,et al.  Pharmaceutical Drug Design Using Dynamic Connectionist Ensemble Networks , 2008, Communications and Discoveries from Multidisciplinary Data.

[30]  G DietterichThomas.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees , 2000 .

[31]  Yihong Gong,et al.  Integrating Document Clustering and Multidocument Summarization , 2011, TKDD.

[32]  Louisa Lam,et al.  Classifier Combinations: Implementations and Theoretical Issues , 2000, Multiple Classifier Systems.

[33]  Fabio Roli,et al.  A Theoretical Analysis of Bagging as a Linear Combination of Classifiers , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Xizhao Wang,et al.  Induction of multiple fuzzy decision trees based on rough set technique , 2008, Inf. Sci..

[35]  Li-Juan Wang,et al.  An improved multiple fuzzy NNC system based on mutual information and fuzzy integral , 2011, Int. J. Mach. Learn. Cybern..

[36]  Fabio Roli,et al.  Methods for dynamic classifier selection , 1999, Proceedings 10th International Conference on Image Analysis and Processing.

[37]  Fabio Roli,et al.  Multiple classifier systems for robust classifier design in adversarial environments , 2010, Int. J. Mach. Learn. Cybern..

[38]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[39]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[41]  Thomas G. Dietterich.  Machine-Learning Research , 1997, AI Mag..

[42]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Vladimir Cherkassky,et al.  Model complexity control for regression using VC generalization bounds , 1999, IEEE Trans. Neural Networks.