Can Neural Networks Do Better Than the Vapnik-Chervonenkis Bounds?

We describe a series of careful numerical experiments which measure the average generalization capability of neural networks trained on a variety of simple functions. These experiments are designed to test whether average generalization performance can surpass the worst-case bounds obtained from formal learning theory using the Vapnik-Chervonenkis dimension (Blumer et al., 1989). We indeed find that, in some cases, the average generalization is significantly better than the VC bound: the approach to perfect performance is exponential in the number of examples m, rather than the 1/m result of the bound. In other cases, we do find the 1/m behavior of the VC bound, and in these cases, the numerical prefactor is closely related to prefactor contained in the bound.