Estimating the number of hidden neurons in recurrent neural networks for nonlinear system identification

The problem of complexity is here addressed by defining an upper bound for the number of the hidden layer's neurons. This majorant is evaluated by applying a singular value decomposition to the contaminated oblique subspace projection of the row space of future outputs into the past inputs-outputs row space, along the future inputs row space. Full rank projections are dealt with by i) computing the number of dominant singular values, on the basis of a threshold related to the Euclidean norm of an artificial error matrix and ii) finding the argument of minimizing the singular value criterion. Results on a benchmark three-tank system demonstrate the effectiveness of the proposed methodology.

[1]  Dietmar Bauer,et al.  Estimating ARMAX systems for multivariate time series using the state approach to subspace algorithms , 2009, J. Multivar. Anal..

[2]  Dietmar Bauer,et al.  Order estimation for subspace methods , 2001, Autom..

[3]  Sabine Van Huffel,et al.  High-performance numerical algorithms and software for subspace-based linear multivariable system identification , 2004 .

[4]  Stephan Trenn,et al.  Multilayer Perceptrons: Approximation Order and Necessary Number of Hidden Units , 2008, IEEE Transactions on Neural Networks.

[5]  Kay Chen Tan,et al.  Estimating the Number of Hidden Neurons in a Feedforward Network Using the Singular Value Decomposition , 2006, IEEE Trans. Neural Networks.

[6]  G. Stewart.  Determining rank in the presence of error , 1992 .

[7]  Antônio de Pádua Braga,et al.  Constructive and pruning methods for neural network design , 2002, VII Brazilian Symposium on Neural Networks, 2002. SBRN 2002. Proceedings..

[8]  Bart De Moor,et al.  Algorithms for Subspace State-Space System Identification: An Overview , 1999 .

[9]  C. Lee Giles,et al.  Overfitting and neural networks: conjugate gradient and backpropagation , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[10]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[11]  Teresa Bernarda Ludermir,et al.  An Optimization Methodology for Neural Network Weights and Architectures , 2006, IEEE Transactions on Neural Networks.

[12]  Bernhard Sendhoff,et al.  Neural network regularization and ensembling using multi-objective evolutionary algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[13]  Wolfgang M. Schmidt,et al.  Diophantine Approximations and Diophantine Equations , 1991 .

[14]  Vladimir Cherkassky,et al.  Model complexity control for regression using VC generalization bounds , 1999, IEEE Trans. Neural Networks.

[15]  G. Stewart,et al.  Rank degeneracy and least squares problems , 1976 .

[16]  Shun-ichi Amari,et al.  Network information criterion-determining the number of hidden units for an artificial neural network model , 1994, IEEE Trans. Neural Networks.