Learning curves from a modified VC-formalism: a case study

In this paper we present a case study of a 1-dimensional higher order neuron using a statistical approach to learning theory which incorporates some information on the distribution on the sample space and can be viewed as a modification of the Vapnik-Chervonenkis formalism (VC-formalism). We concentrate on learning curves defined as averages of the worst generalization error of binary hypothesis consistent with the target on training samples as a function of the training sample size. The true learning curve is derived and compared against estimates from the classical formalism and its modification. It is shown that the modified VC-formalism improves the VC-learning curve by a factor of and also produces a meaningful result for small training sample sizes where VC-bounds are void.

[1]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[2]  Adam Kowalczyk,et al.  Generalisation in Feedforward Networks , 1994, NIPS.

[3]  James A. Pittman,et al.  Recognizing Hand-Printed Letters and Digits Using Backpropagation Learning , 1991, Neural Computation.

[4]  A. Sakurai,et al.  n-h-1 networks store no less n*h+1 examples, but sometimes no more , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[5]  Shun-ichi Amari,et al.  Four Types of Learning Curves , 1992, Neural Computation.

[6]  Yann LeCun,et al.  Measuring the VC-Dimension of a Learning Machine , 1994, Neural Computation.

[7]  Adam Kowalczyk,et al.  Estimates of Storage Capacity of Multilayer Perceptron with Threshold Logic Hidden Units , 1997, Neural Networks.

[8]  Peter L. Bartlett,et al.  Examples of learning curves from a modified VC-formalism , 1995, NIPS.

[9]  D. Haussler,et al.  Rigorous learning curve bounds from statistical mechanics , 1996 .

[10]  David Haussler,et al.  Rigorous Learning Curve Bounds from Statistical Mechanics , 1994, COLT.

[11]  P. Moran,et al.  Random division of an interval , 1951, Mathematical Proceedings of the Cambridge Philosophical Society.

[12]  Vladimir Vapnik.  Estimations of dependences based on statistical data , 1982 .

[13]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..