Stable Feature Selection with Support Vector Machines

The support vector machine (SVM) is a popular method for classification, well known for finding the maximum-margin hyperplane. Combining SVM with \(l_{1}\)-norm penalty further enables it to simultaneously perform feature selection and margin maximization within a single framework. However, \(l_{1}\)-norm SVM shows instability in selecting features in presence of correlated features. We propose a new method to increase the stability of \(l_{1}\)-norm SVM by encouraging similarities between feature weights based on feature correlations, which is captured via a feature covariance matrix. Our proposed method can capture both positive and negative correlations between features. We formulate the model as a convex optimization problem and propose a solution based on alternating minimization. Using both synthetic and real-world datasets, we show that our model achieves better stability and classification accuracy compared to several state-of-the-art regularized classification methods.