Multi-Class Feature Selection with Support Vector Machines

Feature selection is an important component of text categorization that has mostly been addressed by filter methods. In the context of SVM classifiers we propose in this paper a general framework where feature selection is embedded as a part of the learning algorithms. As special cases of this framework we derive two algorithms, one based on the L1 regularization of the weight vector and the other being an extension of the Recursive Feature Elimination algorithm. We also derive a new method that performs even better than these two methods. Our framework and methods are developed in a multi-class setting where the goal is to find a small set of features for all the classes simultaneously. On some datasets the size of the feature set found by our new method is one order of magnitude smaller than the one found by a filter method based on information gain. Finally, we devise a computationally efficient optimization technique for this method.