Sup-norm approximation bounds for networks through probabilistic methods

We consider the problem of approximating a smooth target function and its derivatives by networks involving superpositions and translations of a fixed activation function. The approximation is with respect to the sup-norm and the rate is shown to be of order O(n/sup -1/2/); that is, the rate is independent of the dimension d. The results apply to neural and wavelet networks and extend the work of Barren(see Proc. 7th Yale Workshop on Adaptive and Learning Systems, May, 1992, and ibid., vol.39, p.930, 1993). The approach involves probabilistic methods based on central limit theorems for empirical processes indexed by classes of functions. >

[1]  J. Kuelbs.  Probability on Banach spaces , 1978 .

[2]  R. Dudley.  Universal Donsker Classes and Metric Entropy , 1987 .

[3]  I. Daubechies.  Orthonormal bases of compactly supported wavelets , 1988 .

[4]  B. Irie,et al.  Capabilities of three-layered perceptrons , 1988, IEEE 1988 International Conference on Neural Networks.

[5]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[6]  Michael I. Jordan,et al.  Generic constraints on underspecified target trajectories , 1989, International 1989 Joint Conference on Neural Networks.

[7]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[8]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[9]  R. Gencay,et al.  An algorithm for the n Lyapunov exponents of an n -dimensional unknown dynamical system , 1992 .

[10]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[11]  A. Gallant,et al.  Estimating the Lyapunov Exponent of a Chaotic System with Nonparametric Regression , 1992 .

[12]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[13]  Qinghua Zhang,et al.  Wavelet networks , 1992, IEEE Trans. Neural Networks.

[14]  Eduardo D. Sontag,et al.  Rate of approximation results motivated by robust neural network learning , 1993, COLT '93.

[15]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[16]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[17]  Kurt Hornik,et al.  Some new results on neural network approximation , 1993, Neural Networks.

[18]  S. E. Kelly,et al.  Pointwise convergence of wavelet expansions , 1994, math/9401221.

[19]  Kurt Hornik,et al.  Degree of Approximation Results for Feedforward Networks Approximating Unknown Mappings and Their Derivatives , 1994, Neural Computation.

[20]  Bernard Delyon,et al.  Accuracy analysis for wavelet approximations , 1995, IEEE Trans. Neural Networks.

[21]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .