Feature subset selection for support vector machines through sensitivity analysis

In the context of support vector machines, feature selection is motivated mainly by the consideration of classification speed and generalization ability. Sensitivity analysis of MLP and RBF has already been successfully applied in feature subset selection. We present a novel feature selection method for support vector machines (SVMs) using the sensitivity analysis of SVMs, which is defined as the deviation of separation margin with respect to the perturbation of given feature. The method we proposed can directly be applied to multi-class SVMs. Our experiments validate that the proposed strategy produces satisfactory results both on artificial and real-world data.

[1]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[2]  Pavel Paclík,et al.  Adaptive floating search methods in feature selection , 1999, Pattern Recognit. Lett..

[3]  Chong-Ho Choi,et al.  Sensitivity analysis of multilayer perceptron with differentiable activation functions , 1992, IEEE Trans. Neural Networks.

[4]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[5]  Daniel S. Yeung,et al.  Input sample selection for RBF neural network classification problems using sensitivity measure , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[6]  Andries Petrus Engelbrecht,et al.  A new pruning heuristic based on variance analysis of sensitivity information , 2001, IEEE Trans. Neural Networks.

[7]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[8]  D.S. Yeung,et al.  Input dimensionality reduction for radial basis neural network classification problems using sensitivity measure , 2002, Proceedings. International Conference on Machine Learning and Cybernetics.

[9]  Daniel S. Yeung,et al.  Sensitivity analysis of multilayer perceptron to input and weight perturbations , 2001, IEEE Trans. Neural Networks.

[10]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[11]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[12]  Sayan Mukherjee,et al.  Feature Selection for SVMs , 2000, NIPS.

[13]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[14]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[15]  S. Zenzo.  Pattern recognition: a statistical approach: Devijver P A and Kittler J Prentice-Hall, Englewood Cliffs, NJ, USA (1982) pp 448. £24.95 , 1985 .

[16]  Bernard Widrow,et al.  Sensitivity of feedforward neural networks to weight errors , 1990, IEEE Trans. Neural Networks.

[17]  Chong-Ho Choi,et al.  Input feature selection for classification problems , 2002, IEEE Trans. Neural Networks.