The probably approximately correct (PAC) model of learning from examples is generalized. The problem of learning functions from a set X into a set Y is considered, assuming only that the examples are generated by independent draws according to an unknown probability measure on X*Y. The learner's goal is to find a function in a given hypothesis space of functions from X into Y that on average give Y values that are close to those observed in random examples. The discrepancy is measured by a bounded real-valued loss function. The average loss is called the error of the hypothesis. A theorem on the uniform convergence of empirical error estimates to true error rates is given for certain hypothesis spaces, and it is shown how this implies learnability. A generalized notion of VC dimension that applies to classes of real-valued functions and a notion of capacity for classes of functions that map into a bounded metric space are given. These measures are used to bound the rate of convergence of empirical error estimates to true error rates, giving bounds on the sample size needed for learning using hypotheses in these classes. As an application, a distribution-independent uniform convergence result for certain classes of functions computed by feedforward neural nets is obtained. Distribution-specific uniform convergence results for classes of functions that are uniformly continuous on average are also obtained.<<ETX>>
[1]
Benoit B. Mandelbrot,et al.
Fractal Geometry of Nature
,
1984
.
[2]
R. Dudley.
Central Limit Theorems for Empirical Measures
,
1978
.
[3]
J. Yorke,et al.
Dimension of chaotic attractors
,
1982
.
[4]
J. D. Farmer,et al.
Information Dimension and the Probabilistic Structure of Chaos
,
1982
.
[5]
Leslie G. Valiant,et al.
A theory of the learnable
,
1984,
STOC '84.
[6]
R. Dudley.
A course on empirical processes
,
1984
.
[7]
David Haussler,et al.
ɛ-nets and simplex range queries
,
1987,
Discret. Comput. Geom..
[8]
Emo Welzl,et al.
Partition trees for triangle counting and other range searching problems
,
1988,
SCG '88.
[9]
D. Angluin.
Queries and Concept Learning
,
1988
.
[10]
Alon Itai,et al.
Learnability by fixed distributions
,
1988,
COLT '88.
[11]
Luc Devroye,et al.
Automatic Pattern Recognition: A Study of the Probability of Error
,
1988,
IEEE Trans. Pattern Anal. Mach. Intell..
[12]
B. K. Natarajan,et al.
Some results on learning
,
1989
.
[13]
David Haussler,et al.
Learnability and the Vapnik-Chervonenkis dimension
,
1989,
JACM.