Variables Selection for Multiclass SVM Using the Multiclass Radius Margin Bound

Support vector machines (SVM) are considered as a powerful tool for classification which demonstrate great performances in various fields. Presented for the first time for binary problems, SVMs have been extended in several ways to multiclass case with good results in practice. However, the existence of noise or redundant variables can reduce their performances, where the need for a selection of variables. In this work, we are interested in determining the relevant explanatory variables for an SVM model in the case of multiclass discrimination (MSVM). The criterion proposed here consist in determining such variables using one of the upper bounds of generalization error specific to MSVM models known as radius margin bound [1]. A score derived from this bound will establish the order of relevance of variables, then, the selection of optimal subset will be done using forward method. The experiments are conducted on simulated and real data, and some results are compared with those of other methods of variable selection by MSVM.

[1]  Yi Lin.  Multicategory Support Vector Machines, Theory, and Application to the Classification of . . . , 2003 .

[2]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[3]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[4]  Xiang-Yan Zeng,et al.  Multi-class feature selection for texture classification , 2006, Pattern Recognit. Lett..

[5]  Anis Ben Ishak,et al.  Sélection de variables par les machines à vecteurs supports pour la discrimination binaire et multiclasse en grande dimension , 2007 .

[6]  R Kahavi,et al.  Wrapper for feature subset selection , 1997 .

[7]  Jie Yang,et al.  Feature Selection for Multi-class Problems Using Support Vector Machines , 2004, PRICAI.

[8]  Stephen T. C. Wong,et al.  Multiclass Cancer Classification by Using Fuzzy Support Vector Machine and Binary Decision Tree With Gene Selection , 2005, Journal of biomedicine & biotechnology.

[9]  Paul S. Bradley,et al.  Feature Selection via Concave Minimization and Support Vector Machines , 1998, ICML.

[10]  Emmanuel Monfrini,et al.  A Quadratic Loss Multi-Class SVM for which a Radius-Margin Bound Applies , 2011, Informatica.

[11]  Alain Rakotomamonjy,et al.  Variable Selection Using SVM-based Criteria , 2003, J. Mach. Learn. Res..

[12]  O. Chapelle.  Multi-Class Feature Selection with Support Vector Machines , 2008 .

[13]  H. Zou,et al.  The F ∞ -norm support vector machine , 2008 .

[14]  Pat Langley,et al.  Selection of Relevant Features and Examples in Machine Learning , 1997, Artif. Intell..

[15]  Jian Guo,et al.  Class-specific variable selection for multicategory support vector machines , 2011 .

[16]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[17]  Bernhard Schölkopf,et al.  Use of the Zero-Norm with Linear Models and Kernel Methods , 2003, J. Mach. Learn. Res..

[18]  Yufeng Liu,et al.  Variable Selection via A Combination of the L0 and L1 Penalties , 2007 .

[19]  Juntao Li,et al.  Huberized Multiclass Support Vector Machine for Microarray Classification , 2010 .

[20]  Yann Guermeur,et al.  SVM Multiclasses, Théorie et Applications , 2007 .

[21]  Chih-Chieh Yang,et al.  Multiclass SVM-RFE for product form feature selection , 2008, Expert Syst. Appl..

[22]  Hao Helen Zhang,et al.  Variable selection for the multicategory SVM via adaptive sup-norm regularization , 2008, 0803.3676.

[23]  Koby Crammer,et al.  On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines , 2002, J. Mach. Learn. Res..

[24]  Xin Zhou,et al.  MSVM-RFE: extensions of SVM-RFE for multiclass gene selection on DNA microarray data , 2007, Bioinform..

[25]  Xiaotong Shen,et al.  On L1-Norm Multiclass Support Vector Machines , 2007 .

[26]  Yann Guermeur,et al.  MSVMpack: A Multi-Class Support Vector Machine Package , 2011, J. Mach. Learn. Res..

[27]  Robert Tibshirani,et al.  1-norm Support Vector Machines , 2003, NIPS.

[28]  M. Ringnér,et al.  Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks , 2001, Nature Medicine.

[29]  Pablo M. Granitto,et al.  Feature selection on wide multiclass problems using OVA-RFE , 2010, Inteligencia Artif..

[30]  Jason Weston,et al.  Multi-Class Support Vector Machines , 1998 .

[31]  Jing-Yu Yang,et al.  Optimal discriminant plane for a small number of samples and design method of classifier on the plane , 1991, Pattern Recognit..

[32]  Xiaotong Shen,et al.  MULTI-CATEGORY SUPPORT VECTOR MACHINES, FEATURE SELECTION AND SOLUTION PATH , 2006 .

[33]  Robert P. W. Duin,et al.  Support Vector Data Description , 2004, Machine Learning.