Multilayer Perceptrons: Approximation Order and Necessary Number of Hidden Units

This paper considers the approximation of sufficiently smooth multivariable functions with a multilayer perceptron (MLP). For a given approximation order, explicit formulas for the necessary number of hidden units and its distributions to the hidden layers of the MLP are derived. These formulas depend only on the number of input variables and on the desired approximation order. The concept of approximation order encompasses Kolmogorov-Gabor polynomials or discrete Volterra series, which are widely used in static and dynamic models of nonlinear systems. The results are obtained by considering structural properties of the Taylor polynomials of the function in question and of the MLP function.

[1]  J. Sjöberg.  Neural networks for modelling and control of dynamic systems: M. Nørgaard, O. Ravn, N. K. Poulsen and L. K. Hansen. Springer-Verlag, London Berlin Heidelberg, 2000, pp. xiv+246 , 2004 .

[2]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[3]  C. J. Harris,et al.  Neural Networks for Modelling and Control , 1994 .

[4]  A. Barron.  Approximation and Estimation Bounds for Artificial Neural Networks , 1991, COLT '91.

[5]  N. S. Barnett,et al.  Private communication , 1969 .

[6]  Stephen A. Billings,et al.  Properties of neural networks with applications to modelling non-linear dynamical systems , 1992 .

[7]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[8]  Kay Chen Tan,et al.  Estimating the Number of Hidden Neurons in a Feedforward Network Using the Singular Value Decomposition , 2006, IEEE Transactions on Neural Networks.

[9]  Frank L. Lewis,et al.  Guest Editorial Special Issue on Neural Networks for Feedback Control Systems , 2007, IEEE Transactions on Neural Networks.

[10]  Teresa Bernarda Ludermir,et al.  An Optimization Methodology for Neural Network Weights and Architectures , 2006, IEEE Transactions on Neural Networks.

[11]  S. A. Billings,et al.  Properties of Neural Networks with Applications to Modelling Nonlinear Dynamics , 1991 .

[12]  Andrew R. Barron,et al.  Approximation and estimation bounds for artificial neural networks , 2004, Machine Learning.

[13]  Peter Otto.  Identifikation nichtlinearer Systeme mit Künstlichen Neuronalen Netzen , 1995 .