Input-side training in constructive neural networks based on error scaling and pruning

This paper presents two new modifications to the input-side training in constructive one-hidden-layer feedforward neural networks (FNNs). One is based on scaling of the network output error to which output of a hidden unit is expected to maximally correlate. Results from extensive simulations of many regression problems are then summarized to demonstrate that constructive FNNs generalization capabilities may be significantly improved by the new technique. The second contribution is a proposal for a new criterion for input-side weight pruning. This pruning technique removes redundant input-side weights simultaneously with the network constructive scheme, leading to a smaller network with comparable generalization capabilities. Simulation results are provided to illustrate the effectiveness of the proposed pruning technique.

[1]  Khashayar Khorasani,et al.  An Adaptive Structure Neural Networks with Application to EEG Automatic Seizure Detection , 1996, Neural Networks.

[2]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[3]  Tsu-Chang Lee,et al.  Structure level adaptation for artificial neural networks , 1991 .

[4]  Dhananjay S. Phatak,et al.  Connectivity and performance tradeoffs in the cascade correlation learning architecture , 1994, IEEE Trans. Neural Networks.

[5]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[6]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[7]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[8]  Lutz Prechelt,et al.  Investigation of the CasCor Family of Learning Algorithms , 1997, Neural Networks.

[9]  Timur Ash,et al.  Dynamic node creation in backpropagation networks , 1989 .

[10]  Yoshio Hirose,et al.  Backpropagation algorithm which varies the number of hidden units , 1989, International 1989 Joint Conference on Neural Networks.

[11]  Tariq Samad,et al.  Back propagation with expected source values , 1991, Neural Networks.

[12]  Ehud D. Karnin,et al.  A simple procedure for pruning back-propagation trained neural networks , 1990, IEEE Trans. Neural Networks.

[13]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[14]  Jenq-Neng Hwang,et al.  Regression modeling in back-propagation and projection pursuit learning , 1994, IEEE Trans. Neural Networks.

[15]  Eric B. Bartlett,et al.  Dynamic node architecture learning: An information theoretic approach , 1994, Neural Networks.