Estimating the Number of Hidden Neurons in a Feedforward Network Using the Singular Value Decomposition

In this letter, we attempt to quantify the significance of increasing the number of neurons in the hidden layer of a feedforward neural network architecture using the singular value decomposition (SVD). Through this, we extend some well-known properties of the SVD in evaluating the generalizability of single hidden layer feedforward networks (SLFNs) with respect to the number of hidden layer neurons. The generalization capability of the SLFN is measured by the degree of linear independency of the patterns in hidden layer space, which can be indirectly quantified from the singular values obtained from the SVD, in a postlearning step. A pruning/growing technique based on these singular values is then used to estimate the necessary number of neurons in the hidden layer. More importantly, we describe in detail properties of the SVD in determining the structure of a neural network particularly with respect to the robustness of the selected model

[1]  P. Hansen.  Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion , 1987 .

[2]  A. Laub,et al.  The singular value decomposition: Its computation and some applications , 1980 .

[3]  Martin Fodslette Møller,et al.  A scaled conjugate gradient algorithm for fast supervised learning , 1993, Neural Networks.

[4]  Tong Heng Lee,et al.  Geometrical interpretation and architecture selection of MLP , 2005, IEEE Transactions on Neural Networks.

[5]  S. Tamura,et al.  Capabilities of a three layer feedforward neural network , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[6]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[7]  M. Tateishi,et al.  Determination of the number of redundant hidden units in a three-layered feedforward neural network , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[8]  Lyle H. Ungar,et al.  SVD-NET: an algorithm that automatically selects network structure , 1994, IEEE Trans. Neural Networks.

[9]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[10]  D. Rumelhart,et al.  The effective dimension of the space of hidden units , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[11]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[12]  G. Stewart,et al.  Rank degeneracy and least squares problems , 1976 .

[13]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[14]  Y. F. Huang,et al.  Bounds on number of hidden neurons of multilayer perceptrons in classification and recognition , 1990, IEEE International Symposium on Circuits and Systems.

[15]  Konstantinos Konstantinides,et al.  Statistical analysis of effective singular values in matrix rank determination , 1988, IEEE Trans. Acoust. Speech Signal Process..

[16]  G. Stewart.  Determining rank in the presence of error , 1992 .

[17]  Gregory J. Wolff,et al.  Optimal Brain Surgeon and general network pruning , 1993, IEEE International Conference on Neural Networks.

[18]  M. Hayashi.  A fast algorithm for the hidden units in a multilayer perceptron , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[19]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[20]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[21]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[22]  Panos J. Antsaklis,et al.  A simple method to derive bounds on the size and to train multilayer neural networks , 1991, IEEE Trans. Neural Networks.

[23]  Shin'ichi Tamura,et al.  Capabilities of a four-layered feedforward neural network: four layers versus three , 1997, IEEE Trans. Neural Networks.