Smoothed Bootstrap and Statistical Data Cloning for Classifier Evaluation

This paper is concerned with the estimation of a classifier’s accuracy. We present a number of novel bootstrap estimators, based on kernel smoothing, that consistently show superior performance on both synthetic and real data, with respect to other established methods. We call the process of (re)sampling the data via kernel-based smoothed bootstrap data cloning. The new cloning methods outperform cross-validation and the .632+ bootstrap, which, according to Efron and Tibshirani, is the estimator of choice. Finally, we extend our estimators to complex real-life data sets, in which a data point might include real, bounded, integer and nominal attributes, thus allowing for better classifier evaluation over limited real data repositories such as the UCI repository.

[1]  G. McLachlan.  Discriminant Analysis and Statistical Pattern Recognition , 1992 .

[2]  G. A. Young,et al.  The bootstrap: To smooth or not to smooth? , 1987 .

[3]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[4]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[5]  Charles Elkan,et al.  Estimating the Accuracy of Learned Concepts , 1993, IJCAI.

[6]  Thorsten Joachims,et al.  Estimating the Generalization Performance of an SVM Efficiently , 2000, ICML.

[7]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[8]  Alan M. Polansky,et al.  Smoothed bootstrap confidence intervals with discrete data , 1997 .

[9]  Dana Ron,et al.  Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation , 1997, Neural Computation.

[10]  Thomas J. DiCiccio,et al.  On Smoothing and the Bootstrap , 1989 .

[11]  R. Tibshirani,et al.  Improvements on Cross-Validation: The 632+ Bootstrap Method , 1997 .

[12]  Luc Devroye,et al.  Nonparametric Density Estimation , 1985 .

[13]  L. Devroye,et al.  Nonparametric Density Estimation: The L 1 View. , 1985 .

[14]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[15]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[16]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[17]  Stephen P. Brooks.  Bootstrap methods and their application , 1998 .

[18]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[19]  A. Guillou,et al.  On the smoothed bootstrap , 2000 .

[20]  M. C. Jones,et al.  A Brief Survey of Bandwidth Selection for Density Estimation , 1996 .

[21]  L. Devroye.  Non-Uniform Random Variate Generation , 1986 .

[22]  J. Shao,et al.  The jackknife and bootstrap , 1996 .

[23]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[24]  B. Efron.  Jackknife‐After‐Bootstrap Standard Errors and Influence Functions , 1992 .

[25]  B. Efron.  Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation , 1983 .

[26]  John Langford,et al.  Beating the hold-out: bounds for K-fold and progressive cross-validation , 1999, COLT '99.

[27]  Kohji Fukunaga,et al.  Introduction to Statistical Pattern Recognition-Second Edition , 1990 .

[28]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[29]  J. Aitchison,et al.  Multivariate binary discrimination by the kernel method , 1976 .

[30]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[31]  M. Chavance.  [Jackknife and bootstrap]. , 1992, Revue d'epidemiologie et de sante publique.

[32]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .