Orthogonal Least Squares Algorithm for Training Cascade Neural Networks

This paper proposes a novel constructive training algorithm for cascade neural networks. By reformulating the cascade neural network as a linear-in-the-parameters model, we use the orthogonal least squares (OLS) method to derive a novel objective function for training new hidden units. With this objective function, the sum of squared errors (SSE) of the network can be maximally reduced after each new hidden unit is added, thus leading to a network with less hidden units and better generalization performance. Furthermore, the proposed algorithm considers both the input weights training and output weights training in an integrated framework, which greatly simplifies the training of output weights. The effectiveness of the proposed algorithm is demonstrated by simulation results.

[1]  Tamás D. Gedeon,et al.  A Cascade Network Algorithm Employing Progressive RPROP , 1997, IWANN.

[2]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[3]  Christian O'Reilly,et al.  Analyzing Oscillations for an $N$-node Recurrent Neural Networks Model With Time Delays and General Activation Functions , 2011, IEEE Transactions on Circuits and Systems I: Regular Papers.

[4]  Bing Lam Luk,et al.  Construction of Tunable Radial Basis Function Networks Using Orthogonal Forward Selection , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[5]  Geoffrey E. Hinton,et al.  Learning representations of back-propagation errors , 1986 .

[6]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[7]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[8]  Pierre Courrieu.  A convergent generator of neural networks , 1993, Neural Networks.

[9]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[10]  Osamu Fujita,et al.  Optimization of the hidden unit function in feedforward neural networks , 1992, Neural Networks.

[11]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[12]  B.M. Wilamowski,et al.  Neural network architectures and learning algorithms , 2009, IEEE Industrial Electronics Magazine.

[13]  Hao Yu,et al.  Improved Computation for Levenberg–Marquardt Training , 2010, IEEE Transactions on Neural Networks.

[14]  Xiao Zhi Gao,et al.  Fusion of clonal selection algorithm and differential evolution method in training cascade-correlation neural network , 2009, Neurocomputing.

[15]  Bogdan M. Wilamowski,et al.  Solving parity-N problems with feedforward neural networks , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[16]  Jenq-Neng Hwang,et al.  Regression modeling in back-propagation and projection pursuit learning , 1994, IEEE Trans. Neural Networks.

[17]  Kang Li,et al.  Two-Stage Mixed Discrete–Continuous Identification of Radial Basis Function (RBF) Neural Models for Nonlinear Systems , 2009, IEEE Transactions on Circuits and Systems I: Regular Papers.

[18]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[19]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[20]  Bogdan M. Wilamowski,et al.  Challenges in applications of computational intelligence in industrial electronics , 2010, 2010 IEEE International Symposium on Industrial Electronics.

[21]  Teuvo Kohonen,et al.  The self-organizing map , 1990, Neurocomputing.

[22]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[23]  Sheng Chen,et al.  Identification of nonlinear systems using generalized kernel models , 2005, IEEE Transactions on Control Systems Technology.

[24]  George W. Irwin,et al.  A fast nonlinear model identification method , 2005, IEEE Transactions on Automatic Control.

[25]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[26]  Binghuang Cai,et al.  From Zhang Neural Network to Newton Iteration for Matrix Inversion , 2009, IEEE Transactions on Circuits and Systems I: Regular Papers.

[27]  Kang Li,et al.  Neural input selection - A fast model-based approach , 2007, Neurocomputing.

[28]  De-Shuang Huang,et al.  A Hybrid Forward Algorithm for RBF Neural Network Construction , 2006, IEEE Transactions on Neural Networks.

[29]  K. Lang,et al.  Learning to tell two spirals apart , 1988 .

[30]  Hao Yu,et al.  Neural Network Learning Without Backpropagation , 2010, IEEE Transactions on Neural Networks.

[31]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[32]  Mauro Forti,et al.  Limit Set Dichotomy and Convergence of Cooperative Piecewise Linear Neural Networks , 2011, IEEE Transactions on Circuits and Systems I: Regular Papers.

[33]  Terrence L. Fine,et al.  Forecasting Demand for Electric Power , 1992, NIPS.