Bounds on the number of hidden neurons in multilayer perceptrons

Fundamental issues concerning the capability of multilayer perceptrons with one hidden layer are investigated. The studies are focused on realizations of functions which map from a finite subset of E(n) into E(d). Real-valued and binary-valued functions are considered. In particular, a least upper bound is derived for the number of hidden neurons needed to realize an arbitrary function which maps from a finite subset of E(n ) into E(d). A nontrivial lower bound is also obtained for realizations of injective functions. This result can be applied in studies of pattern recognition and database retrieval. An upper bound is given for realizing binary-valued functions that are related to pattern-classification problems.

[1]  L. Schläfli.  Gesammelte mathematische Abhandlungen , 1950 .

[2]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[3]  Rangasami L. Kashyap,et al.  Synthesis of Switching Functions by Threshold Elements , 1966, IEEE Trans. Electron. Comput..

[4]  R. Winder.  Partitions of N-Space by Hyperplanes , 1966 .

[5]  H. Schwarz.  Gesammelte mathematische Abhandlungen , 1970 .

[6]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[7]  Soo-Young Lee,et al.  An Optimization Network for Matrix Inversion , 1987, NIPS.

[8]  Eric B. Baum,et al.  On the capabilities of multilayer perceptrons , 1988, J. Complex..

[9]  Sverrir Olafsson,et al.  The Capacity of Multilevel Threshold Functions , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Yih-Fang Huang,et al.  Applications and analysis of second order artificial neural networks , 1988, Proceedings of the 27th IEEE Conference on Decision and Control.

[11]  S. Y. Kung,et al.  An algebraic projection analysis for optimal hidden units size and learning rates in back-propagation learning , 1988, IEEE 1988 International Conference on Neural Networks.

[12]  M. Arai.  Mapping abilities of three-layer neural networks , 1989, International 1989 Joint Conference on Neural Networks.

[13]  V. Cherkassky,et al.  Performance of back propagation networks for associative database retrieval , 1989, International 1989 Joint Conference on Neural Networks.

[14]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[15]  Separable Regions.  On Hidden Nodes for Neural Nets , 1989 .

[16]  Y.-F. Huang,et al.  Learning algorithms for perceptions using back-propagation with selective updates , 1990, IEEE Control Systems Magazine.

[17]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..