Approximation of functions by perceptron networks with bounded number of hidden units

Abstract We examine the effect of constraining the number of hidden units. For one-hidden-layer networks with a fairly general type of units (including perceptrons with any bounded activation function and radial-basis-function units), we show that when also the size of parameters is bounded, the best approximation property is satisfied, which means that there always exists a parametrization achieving the global minimum of any error function generated by a supremum orLp-norm. We also show that the only functions that can be approximated with arbitrary accuracy by increasing parameters in networks with a fixed number of Heaviside perceptrons are functions equal almost everywhere to functions that can be exactly computed by such networks. We give a necessary condition on values that such piecewise constant functions must achieve.

[1]  R. DeVore,et al.  Optimal nonlinear approximation , 1989 .

[2]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[3]  H. Mhaskar,et al.  Neural networks for localized approximation , 1994 .

[4]  Charles A. Micchelli,et al.  Dimension-independent bounds on the degree of approximation by neural networks , 1994, IBM J. Res. Dev..

[5]  Richard J. Mammone,et al.  Artificial neural networks for speech and vision , 1994 .

[6]  Eduardo D. Sontag,et al.  Rate of approximation results motivated by robust neural network learning , 1993, COLT '93.

[7]  C. Micchelli,et al.  Approximation by superposition of sigmoidal and radial basis functions , 1992 .

[8]  Kurt Hornik,et al.  Some new results on neural network approximation , 1993, Neural Networks.

[9]  Halbert White,et al.  Approximating and learning unknown mappings using multilayer feedforward networks with bounded weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[10]  Edward K. Blum,et al.  Approximation theory and feedforward networks , 1991, Neural Networks.

[11]  George Finlay Simmons,et al.  Introduction to Topology and Modern Analysis , 1963 .

[12]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[13]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.