Constructing L2-SVM-Based Fuzzy Classifiers in High-Dimensional Space With Automatic Model Selection and Fuzzy Rule Ranking

In this paper, a new scheme for constructing parsimonious fuzzy classifiers is proposed based on the L2-support vector machine (L2-SVM) technique with model selection and feature ranking performed simultaneously in an integrated manner, in which fuzzy rules are optimally generated from data by L2-SVM learning. In order to identify the most influential fuzzy rules induced from the SVM learning, two novel indexes for fuzzy rule ranking are proposed and named as alpha-values and omega-values of fuzzy rules in this paper. The alpha-values are defined as the Lagrangian multipliers of the L2-SVM and adopted to evaluate the output contribution of fuzzy rules, while the omega-values are developed by considering both the rule base structure and the output contribution of fuzzy rules. As a prototype-based classifier, the L2-SVM-based fuzzy classifier evades the curse of dimensionality in high-dimensional space in the sense that the number of support vectors, which equals the number of induced fuzzy rules, is not related to the dimensionality. Experimental results on high-dimensional benchmark problems have shown that by using the proposed scheme the most influential fuzzy rules can be effectively induced and selected, and at the same time feature ranking results can also be obtained to construct parsimonious fuzzy classifiers with better generalization performance than the well-known algorithms in literature.

[1]  Bernhard Schölkopf,et al.  The connection between regularization operators and support vector kernels , 1998, Neural Networks.

[2]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[3]  James C. Bezdek,et al.  Pattern Recognition with Fuzzy Objective Function Algorithms , 1981, Advanced Applications in Pattern Recognition.

[4]  Mark J. L. Orr.  Optimising the widths of radial basis functions , 1998, Proceedings 5th Brazilian Symposium on Neural Networks (Cat. No.98EX209).

[5]  Nikhil R. Pal,et al.  Integrated feature analysis and fuzzy rule-based system identification in a neuro-fuzzy paradigm , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[6]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[7]  Yukihiro Toyoda,et al.  A parameter optimization method for radial basis function type models , 2003, IEEE Trans. Neural Networks.

[8]  D. Dubois,et al.  Operations on fuzzy numbers , 1978 .

[9]  D. Nauck,et al.  Nefclass | a Neuro{fuzzy Approach for the Classification of Data , 1995 .

[10]  John Yen,et al.  Application of statistical information criteria for optimal fuzzy model construction , 1998, IEEE Trans. Fuzzy Syst..

[11]  N. Ancona,et al.  Support vector machines for olfactory signals recognition , 2003 .

[12]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[13]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[14]  L. Breiman.  Arcing classifier (with discussion and a rejoinder by the author) , 1998 .

[15]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[16]  John Yen,et al.  Improving the interpretability of TSK fuzzy models by combining global learning and local learning , 1998, IEEE Trans. Fuzzy Syst..

[17]  James C. Bezdek,et al.  Two soft relatives of learning vector quantization , 1995, Neural Networks.

[18]  J. Chiang,et al.  A new kernel-based fuzzy clustering approach: support vector clustering with cell growing , 2003, IEEE Trans. Fuzzy Syst..

[19]  Yixin Chen,et al.  Support vector learning for fuzzy rule-based classification systems , 2003, IEEE Trans. Fuzzy Syst..

[20]  George W. Irwin,et al.  A hybrid linear/nonlinear training algorithm for feedforward neural networks , 1998, IEEE Trans. Neural Networks.

[21]  Josephine Griffith,et al.  Combining Feature Selection and Neural Networks for Solving Classification Problems , 2001 .

[22]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[23]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[24]  L. Breiman.  Arcing Classifiers , 1998 .

[25]  Bernd Fritzke,et al.  Incremental neuro-fuzzy systems , 1997, Optics & Photonics.

[26]  Francesco Camastra,et al.  A novel kernel method for clustering , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[28]  Nikhil R. Pal,et al.  Some novel classifiers designed using prototypes extracted by a new scheme based on self-organizing feature map , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[29]  Geoffrey J. McLachlan,et al.  Discriminant Analysis and Statistical Pattern Recognition: McLachlan/Discriminant Analysis & Pattern Recog , 2005 .

[30]  John Yen,et al.  Simplifying fuzzy rule-based models using orthogonal transformation methods , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[31]  Sankar K. Pal,et al.  Feature analysis: Neural network and fuzzy set theoretic approaches , 1997, Pattern Recognit..

[32]  Magne Setnes,et al.  Rule-based modeling: precision and transparency , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[33]  Roderick Murray-Smith,et al.  Extending the functional equivalence of radial basis function networks and fuzzy inference systems , 1996, IEEE Trans. Neural Networks.

[34]  G. McLachlan.  Discriminant Analysis and Statistical Pattern Recognition , 1992 .

[35]  Bruce A. Whitehead,et al.  Evolving space-filling curves to distribute radial basis functions over an input space , 1994, IEEE Trans. Neural Networks.

[36]  G. C. Mouzouris,et al.  Designing fuzzy logic systems for uncertain environments using a singular-value-QR decomposition method , 1996, Proceedings of IEEE 5th International Fuzzy Systems.

[37]  Robert Babuska,et al.  Rule base reduction: some comments on the use of orthogonal transforms , 2001, IEEE Trans. Syst. Man Cybern. Syst..

[38]  Dimitar Filev,et al.  Generation of Fuzzy Rules by Mountain Clustering , 1994, J. Intell. Fuzzy Syst..

[39]  John Q. Gan,et al.  An Unsupervised Kernel Based Fuzzy C-Means Clustering Algorithm With Kernel Normalisation , 2004, Int. J. Comput. Intell. Appl..

[40]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[41]  Kok Wai Wong,et al.  A Feature Ranking Algorithm for Fuzzy Modelling Problems , 2003 .

[42]  Bernhard Sendhoff,et al.  Extracting Interpretable Fuzzy Rules from RBF Networks , 2003, Neural Processing Letters.

[43]  Nikhil R. Pal,et al.  A neuro-fuzzy scheme for simultaneous feature selection and fuzzy rule-based classification , 2004, IEEE Transactions on Neural Networks.

[44]  Mark A. Girolami,et al.  Mercer kernel-based clustering in feature space , 2002, IEEE Trans. Neural Networks.

[45]  Sheng-De Wang,et al.  Fuzzy support vector machines , 2002, IEEE Trans. Neural Networks.

[46]  Sheng Chen,et al.  Robust maximum likelihood training of heteroscedastic probabilistic neural networks , 1998, Neural Networks.

[47]  B. John Oommen,et al.  On utilizing search methods to select subspace dimensions for kernel-based nonlinear subspace classifiers , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.