Upper bound on pattern storage in feedforward networks

An upper bound on pattern storage is stated for nonlinear feedforward networks with analytic activation functions, like the multilayer perceptron and radial basis function network. The bound is given in terms of the number of network weights, and applies to networks having any number of output nodes and arbitrary connectivity. Starting from the strict interpolation equations and exact finite degree polynomial models for the hidden units, a straightforward proof by contradiction is developed for the upper bound. Several networks, trained by conjugate gradient, are used to demonstrate the tightness of the bound for random patterns.

[1]  Chuanyi Ji,et al.  Performance and efficiency: recent advances in supervised learning , 1999, Proc. IEEE.

[2]  J. D. Powell,et al.  Radial basis function approximations to polynomials , 1989 .

[3]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[4]  M. Newman,et al.  Interpolation and approximation , 1965 .

[5]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[6]  C. Charalambous,et al.  Conjugate gradient algorithm for efficient training of artifi-cial neural networks , 1990 .

[7]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[8]  Michael T. Manry,et al.  Upper Bound on Pattern Storage in Feedforward Networks , 2007, 2007 International Joint Conference on Neural Networks.

[9]  Jiang Li,et al.  Evaluation and improvement of two training algorithms , 2002, Conference Record of the Thirty-Sixth Asilomar Conference on Signals, Systems and Computers, 2002..

[10]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[11]  Christian Mazza,et al.  On the Storage Capacity of Nonlinear Neural Networks , 1997, Neural Networks.

[12]  W. Murray.  Numerical Methods for Unconstrained Optimization , 1975 .

[13]  Abdelkrim Moussaoui,et al.  Hybrid Hot Strip Rolling Force Prediction using a Bayesian Trained Artificial Neural Network and Analytical Models , 2006 .

[14]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[15]  Yaser S. Abu-Mostafa,et al.  Information capacity of the Hopfield model , 1985, IEEE Trans. Inf. Theory.

[16]  Chuanyi Ji,et al.  Capacity of Two-Layer Feedforward Neural Networks with Binary Weights , 1998, IEEE Trans. Inf. Theory.

[17]  Tae Kim,et al.  New learning factor and testing methods for conjugate gradient training algorithm , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[18]  Michel Cosnard,et al.  Bounds on the Number of Units for Computing Arbitrary Dichotomies by Multilayer Perceptrons , 1994, J. Complex..

[19]  Eric B. Baum,et al.  On the capabilities of multilayer perceptrons , 1988, J. Complex..

[20]  Farid U. Dowla,et al.  Backpropagation Learning for Multilayer Feed-Forward Neural Networks Using the Conjugate Gradient Method , 1991, Int. J. Neural Syst..

[21]  Hélène Paugam-Moisy,et al.  Size of Multilayer Networks for Exact Learning: Analytic Approach , 1996, NIPS.

[22]  Ikuo Matsuba,et al.  New approach to the storage capacity of neural networks using the minimum distance between input patterns , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).