论文信息 - Estimating replicability of classifier learning experiments

Estimating replicability of classifier learning experiments

Replicability of machine learning experiments measures how likely it is that the outcome of one experiment is repeated when performed with a different randomization of the data. In this paper, we present an estimator of replicability of an experiment that is efficient. More precisely, the estimator is unbiased and has lowest variance in the class of estimators formed by a linear combination of outcomes of experiments on a given data set.We gathered empirical data for comparing experiments consisting of different sampling schemes and hypothesis tests. Both factors are shown to have an impact on replicability of experiments. The data suggests that sign tests should not be used due to low replicability. Ranked sum tests show better performance, but the combination of a sorted runs sampling scheme with a t-test gives the most desirable performance judged on Type I and II error and replicability.

Remco R. Bouckaert | R. Bouckaert

[1] J. Ross Quinlan,et al. C4.5: Programs for Machine Learning , 1992 .

[2] Alberto Maria Segre,et al. Programs for Machine Learning , 1994 .

[3] Pat Langley,et al. Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[4] Thomas G. Dietterich. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[5] Catherine Blake,et al. UCI Repository of machine learning databases , 1998 .

[6] Ian H. Witten,et al. Data mining: practical machine learning tools and techniques with Java implementations , 2002, SGMD.

[7] Remco R. Bouckaert,et al. Choosing Between Two Learning Algorithms Based on Calibrated Tests , 2003, ICML.

[8] Steven Salzberg,et al. On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach , 1997, Data Mining and Knowledge Discovery.

[9] Yoshua Bengio,et al. Inference for the Generalization Error , 1999, Machine Learning.