A Survey of old and New Results for the Test Error Estimation of a Classifier

Abstract The estimation of the generalization error of a trained classifier by means of a test set is one of the oldest problems in pattern recognition and machine learning. Despite this problem has been addressed for several decades, it seems that the last word has not been written yet, because new proposals continue to appear in the literature. Our objective is to survey and compare old and new techniques, in terms of quality of the estimation, easiness of use, and rigorousness of the approach, so to understand if the new proposals represent an effective improvement on old ones.

[1]  Chih-Jen Lin,et al.  A tutorial on?-support vector machines , 2005 .

[2]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[3]  Martin Anthony,et al.  Cross-validation for binary classification by real-valued functions: theoretical analysis , 1998, COLT' 98.

[4]  John Langford,et al.  Beating the hold-out: bounds for K-fold and progressive cross-validation , 1999, COLT '99.

[5]  E. B. Wilson.  Probable Inference, the Law of Succession, and Statistical Inference , 1927 .

[6]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[7]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[8]  Olivier Gascuel,et al.  Distribution-free performance bounds with the resubstitution error estimate , 1992, Pattern Recognit. Lett..

[9]  Massimiliano Pontil,et al.  Empirical Bernstein Bounds and Sample-Variance Penalization , 2009, COLT.

[10]  J. Langford.  Tutorial on Practical Prediction Theory for Classification , 2005, J. Mach. Learn. Res..

[11]  L. Brown,et al.  Interval Estimation for a Binomial Proportion , 2001 .

[12]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[13]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[14]  Csaba Szepesvári,et al.  Exploration-exploitation tradeoff using variance estimates in multi-armed bandits , 2009, Theor. Comput. Sci..

[15]  Yishay Mansour,et al.  Generalization Bounds for Decision Trees , 2000, COLT.

[16]  Louis Guttman,et al.  A Distribution-Free Confidence Interval for the Mean , 1948 .

[17]  Davide Anguita,et al.  Test error bounds for classifiers: A survey of old and new results , 2011, 2011 IEEE Symposium on Foundations of Computational Intelligence (FOCI).

[18]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[19]  Aarnout Brombacher,et al.  Probability... , 2009, Qual. Reliab. Eng. Int..

[20]  Christopher J. C. Burges,et al.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998, Data Mining and Knowledge Discovery.

[21]  Davide Anguita,et al.  K-Fold Cross Validation for Error Rate Estimate in Support Vector Machines , 2009, DMIN.

[22]  Peter L. Bartlett,et al.  Model Selection and Error Estimation , 2000, Machine Learning.

[23]  V. Bentkus.  On Hoeffding’s inequalities , 2004, math/0410159.

[24]  Ambuj Tewari,et al.  Sparseness vs Estimating Conditional Probabilities: Some Asymptotic Results , 2007, J. Mach. Learn. Res..

[25]  E. S. Pearson,et al.  THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL , 1934 .

[26]  V. Bentkus.  An Inequality for Large Deviation Probabilities of Sums of Bounded i.i.d. Random Variables , 2001 .

[27]  Augustus De Morgan,et al.  On probable Inference , 2014 .

[28]  Lipo Wang.  Support vector machines : theory and applications , 2005 .

[29]  S. Sathiya Keerthi,et al.  Evaluation of simple performance measures for tuning SVM hyperparameters , 2003, Neurocomputing.

[30]  Luc Devroye,et al.  Distribution-free performance bounds with the resubstitution error estimate (Corresp.) , 1979, IEEE Trans. Inf. Theory.

[31]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[32]  Davide Anguita,et al.  Theoretical and Practical Model Selection Methods for Support Vector Classifiers , 2004 .

[33]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[34]  Davide Anguita,et al.  Maximal Discrepancy for Support Vector Machines , 2011, ESANN.