Integrated Analytic Framework for Neural Network Construction

This paper investigates the construction of a wide class of singlehidden layer neural networks (SLNNs) with or without tunable parameters in the hidden nodes. It is a challenging problem if both the parameter training and determination of network size are considered simultaneously. Two alternative network construction methods are considered in this paper. Firstly, the discrete construction of SLNNs is introduced. The main objective is to select a subset of hidden nodes from a pool of candidates with parameters fixed `a priori'. This is called discrete construction since there are no parameters in the hidden nodes that need to be trained. The second approach is called continuous construction as all the adjustable network parameters are trained on the whole parameter space along the network construction process. In the second approach, there is no need to generate a pool of candidates, and the network grows one by one with the adjustable parameters optimized. The main contribution of this paper is to show that the network construction can be done using the above two alternative approaches, and these two approaches can be integrated within a unified analytic framework, leading to potentially significantly improved model performance and/or computational efficiency.

[1]  Narasimhan Sundararajan,et al.  A generalized growing and pruning RBF (GGAP-RBF) neural network for function approximation , 2005, IEEE Transactions on Neural Networks.

[2]  De-Shuang Huang,et al.  A Hybrid Forward Algorithm for RBF Neural Network Construction , 2006, IEEE Transactions on Neural Networks.

[3]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[4]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[5]  George W. Irwin,et al.  A fast nonlinear model identification method , 2005, IEEE Transactions on Automatic Control.

[6]  S. Billings,et al.  Fast orthogonal identification of nonlinear stochastic models and radial basis function neural networks , 1996 .

[7]  Marc Toussaint,et al.  Extracting Motion Primitives from Natural Handwriting Data , 2006, ICANN.

[8]  Minrui Fei,et al.  Real-Time Construction of Neural Networks , 2006, ICANN.

[9]  H. Akaike.  A new look at the statistical model identification , 1974 .

[10]  Michael J. Korenberg,et al.  Iterative fast orthogonal search algorithm for MDL-based training of generalized single-layer networks , 2000, Neural Networks.

[11]  Chee Kheong Siew,et al.  Can threshold networks be trained directly? , 2006, IEEE Transactions on Circuits and Systems II: Express Briefs.

[12]  Kang Li,et al.  A two-stage algorithm for identification of nonlinear dynamic systems , 2006, Autom..

[13]  Hitoshi Iba,et al.  Learning polynomial feedforward neural networks by genetic programming and backpropagation , 2003, IEEE Trans. Neural Networks.

[14]  F.J. Von Zuben,et al.  Hierarchical evolution of heterogeneous neural networks , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[15]  K.M. Adeney,et al.  On the use of separable Volterra networks to model discrete-time Volterra systems , 2001, IEEE Trans. Neural Networks.

[16]  Stephen A. Billings,et al.  International Journal of Control , 2004 .

[17]  Yoh-Han Pao,et al.  Additional perspectives on feedforward neural-nets and the functional-link , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).