An improved multiple fuzzy NNC system based on mutual information and fuzzy integral

Multiple nearest neighbor classifier system (MNNCS) is a popular method to relax the curse of dimensionality. In previous work, most of the MNNCSs are designed by random methods. Random methods may generate unstable component classifiers. In order to relax the randomness, large amount of component classifiers are needed. This paper first extends nearest neighbor classifier into fuzzy nearest neighbor classifier, and proposes a new multiple fuzzy nearest neighbor classifier system based on mutual information and fuzzy integral, called MIFI-MFNNCS. MIFI-MFNNCS adopts target perturbation. Target perturbation decomposes the original classification problem into several sub-problems, where one sub-problem represents one class data. Each sub-problem is described by the relevant data and features. Then it is classified by one component classifier. Therefore, the number of component classifiers can be fixed and reduced. For one component classifier, data may be selected according to its class. And feature is needed to be selected by mutual information. Mutual information can reduce the uncertainty of each component classifier. Feature selection by mutual information in MIFI-MFNNCS may be less affected by the interaction among different classes. The diversity decisions from sub-problem classifiers are combined by fuzzy integral to get the final decision. Here we propose a new method to compute density value according to mutual information, which is a simple method. To demonstrate the performance of the proposed MIFI-MFNNCS, we perform experimental comparisons using five UCI datasets. The results of component classifiers in MIFI-MFNNCS for Ionosphere are shown and analyzed. MIFI-MFNNCS is compared with (1) NNC (2) NNC after feature selection by mutual information (MI-FS-NNC). In multiple fuzzy nearest neighbor classifier system (MFNNCS), mutual information is compared with attribute bagging. And three combination methods are compared, including fuzzy integral, majority voting rule and average. The experimental results show that the accuracy of MIFI-MFNNCS is better than other methods. And mutual information is superior to attribute bagging. Fuzzy integral shows a better performance than majority voting rule and average.

[1]  Zhi-Hua Zhou,et al.  Ensembling local learners ThroughMultimodal perturbation , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[3]  Yang Yu,et al.  Ensembling local learners ThroughMultimodal perturbation , 2005, IEEE Trans. Syst. Man Cybern. Part B.

[4]  Pat Langley,et al.  Average-Case Analysis of a Nearest Neighbor Algorithm , 1993, IJCAI.

[5]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[6]  Tommy W. S. Chow,et al.  Estimating optimal feature subsets using efficient estimation of high-dimensional mutual information , 2005, IEEE Transactions on Neural Networks.

[7]  Chong-Ho Choi,et al.  Input feature selection for classification problems , 2002, IEEE Trans. Neural Networks.

[8]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[9]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[10]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[11]  Y. Wang,et al.  Evaluation Criteria Based on Mutual Information for Classifications Including Rejected Class , 2008 .

[12]  Michel Verleysen,et al.  Resampling methods for parameter-free and robust feature selection with mutual information , 2007, Neurocomputing.

[13]  Kwong-Sak Leung,et al.  A genetic algorithm for determining nonadditive set functions in information fusion , 1999, Fuzzy Sets Syst..

[14]  M. Sugeno.  FUZZY MEASURES AND FUZZY INTEGRALS—A SURVEY , 1993 .

[15]  胡包钢,et al.  关于互信息准则在分类(包括拒识类别)问题中的应用 , 2009 .

[16]  G. Klir,et al.  Fuzzy Measure Theory , 1993 .

[17]  Nicolás García-Pedrajas,et al.  Boosting k-nearest neighbor classifier by means of input space projection , 2009, Expert Syst. Appl..

[18]  James M. Keller,et al.  Training the fuzzy integral , 1996, Int. J. Approx. Reason..

[19]  Qing-Cai Chen,et al.  Combination of Multiple K-NNCS by Fuzzy Integral , 2006, 2006 International Conference on Machine Learning and Cybernetics.

[20]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Stephen D. Bay.  Combining Nearest Neighbor Classifiers Through Multiple Feature Subsets , 1998, ICML.

[22]  Xiaoming Xu,et al.  A hybrid genetic algorithm for feature selection wrapper based on mutual information , 2007, Pattern Recognit. Lett..

[23]  Francis K. H. Quek,et al.  Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets , 2003, Pattern Recognit..

[24]  Mübeccel Demirekler,et al.  On output independence and complementariness in rank-based multiple classifier decision systems , 2001, Pattern Recognit..

[25]  Chong-Ho Choi,et al.  Input Feature Selection by Mutual Information Based on Parzen Window , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Lei Liu,et al.  Feature selection with dynamic mutual information , 2009, Pattern Recognit..

[27]  Jia Wang,et al.  Using Neural Networks to Determine Sugeno Measures by Statistics , 1997, Neural Networks.

[28]  P. Langley,et al.  Average-case analysis of a nearest neighbor algorthim , 1993, IJCAI 1993.