On the Learnability of Rich Function Classes

The probably approximately correct (PAC) model of learning and its extension to real-valued function classes sets a rigorous framework based upon which the complexity of learning a target from a function class using a finite sample can be computed. There is one main restriction, however, that the function class have a finite VC-dimension or scale-sensitive pseudo-dimension. In this paper we present an extension of the PAC framework with which rich function classes with possibly infinite pseudo-dimension may be learned with a finite number of examples and a finite amount of partial information. As an example we consider learning a family of infinite dimensional Sobolev classes.

[1]  M. Birman,et al.  PIECEWISE-POLYNOMIAL APPROXIMATIONS OF FUNCTIONS OF THE CLASSES $ W_{p}^{\alpha}$ , 1967 .

[2]  Russell Greiner,et al.  Computational learning theory and natural learning systems , 1997 .

[3]  David Haussler,et al.  Sphere Packing Numbers for Subsets of the Boolean n-Cube with Bounded Vapnik-Chervonenkis Dimension , 1995, J. Comb. Theory, Ser. A.

[4]  D. Pollard.  Convergence of stochastic processes , 1984 .

[5]  Brian D. Ripley,et al.  Pattern Recognition and Neural Networks , 1996 .

[6]  A. Barron.  Approximation and Estimation Bounds for Artificial Neural Networks , 1991, COLT '91.

[7]  Henryk Wozniakowski,et al.  Information-based complexity , 1987, Nature.

[8]  Y. Abu-Mostafa.  Machines that Learn from Hints , 1995 .

[9]  A. Pinkus.  n-Widths in Approximation Theory , 1985 .

[10]  H. Woxniakowski.  Information-Based Complexity , 1988 .

[11]  Jude W. Shavlik,et al.  Interpretation of Artificial Neural Networks: Mapping Knowledge-Based Neural Networks into Rules , 1991, NIPS.

[12]  Volker Tresp,et al.  Incorporating prior knowledge into networks of locally-tuned units , 1994, COLT 1994.

[13]  G. Lugosi,et al.  Adaptive Model Selection Using Empirical Complexities , 1998 .

[14]  Yaser S. Abu-Mostafa,et al.  Hints and the VC Dimension , 1993, Neural Computation.

[15]  Joel Ratsaby,et al.  The Degree of Approximation of Sets in Euclidean Space Using Sets with Bounded Vapnik-Chervonenkis Dimension , 1998, Discret. Appl. Math..

[16]  Noga Alon,et al.  Scale-sensitive dimensions, uniform convergence, and learnability , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[17]  John Shawe-Taylor,et al.  A framework for structural risk minimisation , 1996, COLT '96.

[18]  Yaser S. Abu-Mostafa,et al.  Learning from hints in neural networks , 1990, J. Complex..

[19]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[20]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[21]  G. Lorentz,et al.  Constructive approximation : advanced problems , 1996 .

[22]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[23]  Noga Alon,et al.  Scale-sensitive dimensions, uniform convergence, and learnability , 1997, JACM.

[24]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[25]  Halbert White,et al.  Estimation, inference, and specification analysis , 1996 .

[26]  Joel Ratsaby,et al.  Generalization of the PAC-Model for Learning with Partial Information , 1997, EuroCOLT.

[27]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[28]  D. Pollard.  Empirical Processes: Theory and Applications , 1990 .

[29]  Robert A. Jacobs,et al.  Methods For Combining Experts' Probability Assessments , 1995, Neural Computation.

[30]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[31]  Gábor Lugosi,et al.  Concept learning using complexity regularization , 1995, IEEE Trans. Inf. Theory.

[32]  David Haussler,et al.  Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications , 1992, Inf. Comput..

[33]  Ron Meir,et al.  Towards robust model selection using estimation and approximation error bounds , 1996, COLT '96.

[34]  Joel Ratsaby,et al.  On the Degree of Approximation by Manifolds of Finite Pseudo-Dimension , 1999 .