AbstractFor a common class of artificial neural networks, the mean integrated squared error between the estimated network and a target functionf is shown to be bounded by
$$O\left( {\frac{{\mathop c\nolimits_f^2 }}{n}} \right) + O\left( {\frac{{nd}}{N}\log N} \right)$$
wheren is the number of nodes,d is the input dimension of the function,N is the number of training observations, andCf is the first absolute moment of the Fourier magnitude distribution off. The two contributions to this total risk are the approximation error and the estimation error. Approximation error refers to the distance between the target function and the closest neural network function of a given architecture and estimation error refers to the distance between this ideal network function and an estimated network function. Withn ∼ Cf(N/(d logN))1/2 nodes, the order of the bound on the mean integrated squared error is optimized to beO(Cf((d/N) logN)1/2). The bound demonstrates surprisingly favorable properties of network estimation compared to traditional series and nonparametric curve estimation techniques in the case thatd is moderately large. Similar bounds are obtained when the number of nodesn is not preselected as a function ofCf (which is generally not knowna priori), but rather the number of nodes is optimized from the observed data by the use of a complexity regularization or minimum description length criterion. The analysis involves Fourier techniques for the approximation error, metric entropy considerations for the estimation error, and a calculation of the index of resolvability of minimum complexity estimation of the family of networks.
[1]
C. J. Stone,et al.
OPTIMAL GLOBAL RATES OF CONVERGENCE FOR NONPARAMETRIC ESTIMATORS
,
1982
.
[2]
D. Cox.
Approximation of Least Squares Regression on Nested Subspaces
,
1988
.
[3]
A. Barron,et al.
Statistical properties of artificial neural networks
,
1989,
Proceedings of the 28th IEEE Conference on Decision and Control,.
[4]
A. Barron.
Approximation and Estimation Bounds for Artificial Neural Networks
,
1991,
COLT '91.
[5]
A. Barron,et al.
APPROXIMATION OF DENSITY FUNCTIONS BY SEQUENCES OF EXPONENTIAL FAMILIES
,
1991
.
[6]
Andrew R. Barron,et al.
Minimum complexity density estimation
,
1991,
IEEE Trans. Inf. Theory.
[7]
L. Jones.
A Simple Lemma on Greedy Approximation in Hilbert Space and Convergence Rates for Projection Pursuit Regression and Neural Network Training
,
1992
.
[8]
David Haussler,et al.
Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications
,
1992,
Inf. Comput..
[9]
Andrew R. Barron,et al.
Universal approximation bounds for superpositions of a sigmoidal function
,
1993,
IEEE Trans. Inf. Theory.
[10]
Daniel F. McCaffrey,et al.
Convergence rates for single hidden layer feedforward networks
,
1994,
Neural Networks.