K-Fold Cross Validation for Error Rate Estimate in Support Vector Machines

In this paper, we review the k–Fold Cross Validation (KCV) technique, applied to the Support Vector Machine (SVM) classification algorithm. We compare several variations on the KCV technique: some of them are often used by practitioners, but without any theoretical justification, while others are less used but more rigorous in finding a correct classifier. The last ones allow to establish an upper– bound of the error rate of the SVM, which represent a way to guarantee, in a statistical sense, the reliability of the classifier and, therefore, turns out to be quite important in many real–world applications. Some experimental results on well–known benchmarking datasets allow to perform the comparison and support our claims.

[1]  R. Tibshirani,et al.  Improvements on Cross-Validation: The 632+ Bootstrap Method , 1997 .

[2]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[3]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[4]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[5]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[6]  D. Anguita,et al.  K-fold generalization capability assessment for support vector classifiers , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[7]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[8]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[9]  Bernhard Schölkopf,et al.  Comparing support vector machines with Gaussian kernels to radial basis function classifiers , 1997, IEEE Trans. Signal Process..

[10]  M. Kaariainen,et al.  Semi-Supervised Model Selection Based on Cross-Validation , 2006 .

[11]  Nello Cristianini,et al.  Margin Distribution and Soft Margin , 2000 .

[12]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[13]  S. Sathiya Keerthi,et al.  Evaluation of simple performance measures for tuning SVM hyperparameters , 2003, Neurocomputing.

[14]  Lipo Wang.  Support vector machines : theory and applications , 2005 .

[15]  John Langford,et al.  A comparison of tight generalization error bounds , 2005, ICML '05.

[16]  John Langford,et al.  Beating the hold-out: bounds for K-fold and progressive cross-validation , 1999, COLT '99.

[17]  Davide Anguita,et al.  Evaluating the Generalization Ability of Support Vector Machines through the Bootstrap , 2000, Neural Processing Letters.

[18]  Davide Anguita,et al.  Theoretical and Practical Model Selection Methods for Support Vector Classifiers , 2004 .

[19]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[20]  Marcos M. Campos,et al.  SVM in Oracle Database 10g: Removing the Barriers to Widespread Adoption of Support Vector Machines , 2005, VLDB.

[21]  Kazuoki Azuma.  WEIGHTED SUMS OF CERTAIN DEPENDENT RANDOM VARIABLES , 1967 .

[22]  Martin Anthony,et al.  Cross-validation for binary classification by real-valued functions: theoretical analysis , 1998, COLT' 98.

[23]  Chih-Jen Lin,et al.  A tutorial on?-support vector machines , 2005 .

[24]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[25]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.