Efron (1983) proposed the 632 bootstrap error rate estimator, and demonstrated its good performance in simulations. However, further recent studies have suggested that it only performs well for a restricted range of true error rates. We conduced an intensive, simulation study of the 632 estimator to investigate this hypothesis in detail. There was no evidence to support the contention, although the estimator's performance did vary with the true error rate.We also investigated some more general aspects of error rate studies. Error rate is bounded by 0 and 1, so that comparisons based on an untransformed scale may be inappropriate. We therefore explored the consequences of stretching the error rate scale. Furthermore, the results of such studies are typically generated as mixture distributions, because the performance results are averages over the true error rate values arising from underlying distributions which have a fixed optimal error rate. We discovered, rather surprisingly, that this has little effect if the conclusions are based on mean squared error.
[1]
G. McLachlan.
Error Rate Estimation in Discriminant Analysis: Recent Advances
,
1987
.
[2]
Abdelmonem A. Afifi,et al.
Comparison of Stopping Rules in Forward Stepwise Discriminant Analysis
,
1979
.
[3]
B. Efron.
Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation
,
1983
.
[4]
D. J. Hand,et al.
Recent advances in error rate estimation
,
1986,
Pattern Recognit. Lett..
[5]
Sangit Chatterjee,et al.
Estimation of misclassification probabilities by bootstrap methods
,
1983
.
[6]
Godfried T. Toussaint,et al.
Bibliography on estimation of misclassification
,
1974,
IEEE Trans. Inf. Theory.