On the stability and bias-variance analysis of sparse SVMs

Stability and bias-variance analysis are two powerful tools to better understand learning algorithms. We use these tools to analyze a class of support vector machines (SVMs) that try to reduce classifier complexity. The motivation for doing this is to compare the original and modified SVMs on two behavioral dimensions (a) stability and (b) learning behavior. Our preliminary experimental results show that (i) the class of algorithms which reduce classifier complexity by reducing the number of support vectors (SVs) are potentially unstable and (ii) the learning behavior is quite similar to the original SVMs.

[1]  Giorgio Valentini,et al.  Bias-Variance Analysis of Support Vector Machines for the Development of SVM-Based Ensemble Methods , 2004, J. Mach. Learn. Res..

[2]  S. Sathiya Keerthi,et al.  Building Support Vector Machines with Reduced Classifier Complexity , 2006, J. Mach. Learn. Res..

[3]  Ioannis Pitas,et al.  Demonstrating the stability of support vector machines for classification , 2006, Signal Process..

[4]  André Elisseeff,et al.  Stability and Generalization , 2002, J. Mach. Learn. Res..

[5]  Yuh-Jye Lee,et al.  RSVM: Reduced Support Vector Machines , 2001, SDM.

[6]  Olvi L. Mangasarian,et al.  Generalized Support Vector Machines , 1998 .

[7]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[9]  Christopher J. C. Burges,et al.  Simplified Support Vector Decision Rules , 1996, ICML.

[10]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[11]  Tong Tang,et al.  Proceedings of the European Symposium on Artificial Neural Networks , 2006 .

[12]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[13]  Ingo Steinwart,et al.  Sparseness of Support Vector Machines , 2003, J. Mach. Learn. Res..

[14]  Michael I. Jordan,et al.  Predictive low-rank decomposition for kernel methods , 2005, ICML.

[15]  Harish Karnick,et al.  Kernel-based online machine learning and support vector reduction , 2008, ESANN.