Pseudo-dimension and entropy of manifolds formed by affine-invariant dictionary

Abstract We consider the manifolds Hn(φ) formed by all possible linear combinations of n functions from the set {φ(A⋅+b)}, where x→Ax+b is arbitrary affine mapping in the space ℝd. For example, neural networks and radial basis functions are the manifolds of type Hn(φ). We obtain estimates for pseudo-dimension of the manifold Hn(φ) for wide collection of the generator function φ. The estimates have the order O(d2n) in degree scale, that is the order is proportional to number of parameters of the manifold Hn(φ). Moreover the estimates for ɛ-entropy of the manifold Hn(φ) are obtained.

[1]  Vitaly Maiorov,et al.  Approximation by neural networks and learning theory , 2006, J. Complex..

[2]  S. Mendelson,et al.  Entropy and the combinatorial dimension , 2002, math/0203275.

[3]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[4]  Marek Karpinski,et al.  Polynomial bounds for VC dimension of sigmoidal neural networks , 1995, STOC '95.

[5]  S. Smale.  Mathematical problems for the next century , 1998 .

[6]  H. Warren.  Lower bounds for approximation by nonlinear manifolds , 1968 .

[7]  N. Dyn,et al.  Multivariate Approximation and Applications: Index , 2001 .

[8]  Allan Pinkus,et al.  Lower bounds for approximation by MLP neural networks , 1999, Neurocomputing.

[9]  Michel Talagrand,et al.  Vapnik--Chervonenkis type conditions and uniform Donsker classes of functions , 2003 .

[10]  Peter L. Bartlett,et al.  Neural Network Learning - Theoretical Foundations , 1999 .

[11]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization , 1997 .

[12]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[13]  J. Milnor.  On the Betti numbers of real varieties , 1964 .

[14]  V. Vapnik,et al.  Necessary and Sufficient Conditions for the Uniform Convergence of Means to their Expectations , 1982 .

[15]  David Haussler,et al.  Sphere Packing Numbers for Subsets of the Boolean n-Cube with Bounded Vapnik-Chervonenkis Dimension , 1995, J. Comb. Theory, Ser. A.

[16]  Ron Meir,et al.  Lower bounds for multivariate approximation by affine-invariant dictionaries , 2001, IEEE Trans. Inf. Theory.

[17]  M. Buhmann.  Multivariate Approximation and Applications: Approximation and interpolation with radial functions , 2001 .

[18]  Peter L. Bartlett,et al.  Almost Linear VC-Dimension Bounds for Piecewise Polynomial Networks , 1998, Neural Computation.