Two-stage extreme learning machine for regression

Extreme learning machine (ELM) proposed by Huang et al. was developed for generalized single hidden layer feedforward networks (SLFNs) with a wide variety of hidden nodes. It proved to be very fast and effective especially for solving function approximation problems with a predetermined network structure. However, the method for determining the network structure of preliminary ELM may be tedious and may not lead to a parsimonious solution. In this paper, a systematic two-stage algorithm (named TS-ELM) is introduced to handle the problem. In the first stage, a forward recursive algorithm is applied to select the hidden nodes from the candidates randomly generated in each step and add them to the network until the stopping criterion achieves its minimum. The significance of each hidden node is then reviewed in the second stage and the insignificance ones are removed from the network, which drastically reduces the network complexity. The effectiveness of TS-ELM is verified by the empirical studies in this paper.

[1]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[2]  George W. Irwin,et al.  A fast nonlinear model identification method , 2005, IEEE Transactions on Automatic Control.

[3]  S. Billings,et al.  Algorithms for minimal model structure detection in nonlinear dynamic system identification , 1997 .

[4]  Kang Li,et al.  A two-stage algorithm for identification of nonlinear dynamic systems , 2006, Autom..

[5]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[6]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[7]  Mauro Birattari,et al.  Recursive Lazy Learning for Modeling and Control , 1998, ECML.

[8]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[9]  Shuzhi Sam Ge,et al.  Fast Construction of Single-Hidden-Layer Feedforward Networks , 2012, Handbook of Natural Computing.

[10]  R. H. Myers.  Classical and modern regression with applications , 1986 .

[11]  S. Chen,et al.  Fast orthogonal least squares algorithm for efficient subset model selection , 1995, IEEE Trans. Signal Process..

[12]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[13]  Guang-Bin Huang,et al.  Classification ability of single hidden layer feedforward neural networks , 2000, IEEE Trans. Neural Networks Learn. Syst..

[14]  Amaury Lendasse,et al.  OP-ELM: Theory, Experiments and a Toolbox , 2008, ICANN.

[15]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[16]  Guang-Bin Huang,et al.  Neuron selection for RBF neural network classifier based on data structure preserving criterion , 2005, IEEE Transactions on Neural Networks.

[17]  S. Billings,et al.  Fast orthogonal identification of nonlinear stochastic models and radial basis function neural networks , 1996 .

[18]  De-Shuang Huang,et al.  A Hybrid Forward Algorithm for RBF Neural Network Construction , 2006, IEEE Transactions on Neural Networks.

[19]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[20]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[21]  Stephen P. Boyd,et al.  Linear Matrix Inequalities in Systems and Control Theory , 1994 .

[22]  Colin L. Mallows,et al.  Some Comments on Cp , 2000, Technometrics.

[23]  Pingkang Li,et al.  Simple recursive algorithm for linear-in-theparameters nonlinear model identification , 2009, Science in China Series F: Information Sciences.

[24]  D. Serre.  Matrices: Theory and Applications , 2002 .

[25]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[26]  H. Akaike.  A new look at the statistical model identification , 1974 .

[27]  Zexuan Zhu,et al.  A fast pruned-extreme learning machine for classification problem , 2008, Neurocomputing.

[28]  Sheng Chen,et al.  Model selection approaches for non-linear system identification: a review , 2008, Int. J. Syst. Sci..

[29]  Yuan Lan,et al.  Constructive hidden nodes selection of extreme learning machine for regression , 2010, Neurocomputing.

[30]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[31]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[32]  Timo Similä,et al.  Multiresponse Sparse Regression with Application to Multidimensional Scaling , 2005, ICANN.

[33]  H. Akaike.  Statistical predictor identification , 1970 .

[34]  Robert F. Stengel,et al.  Smooth function approximation using neural networks , 2005, IEEE Transactions on Neural Networks.

[35]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[36]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[37]  E. Yaz.  Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.