For many learning algorithms, their learning accuracy will increase as the size of training data increases, forming the well-known learning curve. Usually a learning curve can be fitted by interpolating or extrapolating some points on it with a specified model. The obtained learning curve can then be used to predict the maximum achievable learning accuracy or to estimate the amount of data needed to achieve an expected learning accuracy, both of which will be especially meaningful to data mining on large data sets. Although some models have been proposed to model learning curves, most of them do not test their applicability to large data sets. In this paper, we focus on this issue. We empirically compare six potentially useful models by fitting learning curves of two typical classification algorithms--C4.5 (decision tree) and LOG (logistic discrimination) on eight large UCI benchmark data sets. By using all available data for learning, we fit a full-length learning curve; by using a small portion of the data, we fit a part-length learning curve. The models are then compared in terms of two performances: (1) how well they fit a full-length learning curve, and (2) how well a fitted part-length learning curve can predict learning accuracy at the full length. Experimental results show that the power law (y = a - b * x-c) is the best among the six models in both the performances for the two algorithms and all the data sets. These results support the applicability of learning curves to data mining.
[1]
Douglas H. Fisher,et al.
Modeling decision tree performance with the power law
,
1999,
AISTATS.
[2]
Carl M. Kadie,et al.
SEER: maximum likelihood regression for learning-speed curves
,
1995
.
[3]
J. Ross Quinlan,et al.
C4.5: Programs for Machine Learning
,
1992
.
[4]
David J. Spiegelhalter,et al.
Machine Learning, Neural and Statistical Classification
,
2009
.
[5]
Corinna Cortes,et al.
Prediction of Generalization Ability in Learning Machines
,
1994
.
[6]
Catherine Blake,et al.
UCI Repository of machine learning databases
,
1998
.
[7]
Shun-ichi Amari,et al.
Four Types of Learning Curves
,
1992,
Neural Computation.
[8]
David A. Ratkowsky,et al.
Handbook of nonlinear regression models
,
1990
.
[9]
Tim Oates,et al.
Efficient progressive sampling
,
1999,
KDD '99.
[10]
T. Brubaker,et al.
Nonlinear Parameter Estimation
,
1979
.
[11]
Haruhisa Takahashi,et al.
Exponential or Polynomial Learning Curves? Case-Based Studies
,
2000,
Neural Computation.
[12]
Pat Langley,et al.
Static Versus Dynamic Sampling for Data Mining
,
1996,
KDD.