Parsimonious regularized extreme learning machine based on orthogonal transformation

Recently, two parsimonious algorithms were proposed to sparsify extreme learning machine (ELM), i.e., constructive parsimonious ELM (CP-ELM) and destructive parsimonious ELM (DP-ELM). In this paper, the ideas behind CP-ELM and DP-ELM are extended to the regularized ELM (RELM), thus obtaining CP-RELM and DP-RELM. For CP-RELM(DP-RELM), there are two schemes to realize it, viz. CP-RELM-I and CP-RELM-II(DP-RELM-I and DP-RELM-II). Generally speaking, CP-RELM-II(DP-RELM-II) outperforms CP-RELM-I(DP-RELM-I) in terms of parsimoniousness. Under nearly the same generalization, compared with CP-ELM(DP-ELM), CP-RELM-II(DP-RELM-II) usually needs fewer hidden nodes. In addition, different from CP-ELM and DP-ELM, for CP-RELM and DP-RELM the number of candidate hidden nodes may be larger than the number of training samples, which assists the selection of much better hidden nodes for constructing more compact networks. Finally, eleven benchmark data sets divided into two groups are utilized to do experiments and the usefulness of the proposed algorithms is reported.

[1]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.

[2]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[3]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[4]  L. BartlettP..  The sample complexity of pattern classification with neural networks , 2006 .

[5]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[6]  Wei Jiang,et al.  Fast sparse approximation of extreme learning machine , 2014, Neurocomputing.

[7]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[8]  Sheng Chen,et al.  A New RBF Neural Network With Boundary Value Constraints , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[9]  David G. Stork,et al.  Pattern Classification , 1973 .

[10]  Yonggwan Won,et al.  Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks , 2011, Pattern Recognit. Lett..

[11]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[12]  Zhenxing Qian,et al.  Evolutionary selection extreme learning machine optimization for regression , 2012, Soft Comput..

[13]  Amaury Lendasse,et al.  TROP-ELM: A double-regularized ELM using LARS and Tikhonov regularization , 2011, Neurocomputing.

[14]  Dong Sun Park,et al.  Online sequential extreme learning machine with forgetting mechanism , 2012, Neurocomputing.

[15]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[16]  Sundaram Suresh,et al.  Fast learning Circular Complex-valued Extreme Learning Machine (CC-ELM) for real-valued classification problems , 2012, Inf. Sci..

[17]  Hongming Zhou,et al.  Optimization method based extreme learning machine for classification , 2010, Neurocomputing.

[18]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[19]  Wentao Mao,et al.  Model selection of extreme learning machine based on multi-objective optimization , 2011, Neural Computing and Applications.

[20]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[21]  Qinyu. Zhu.  Extreme Learning Machine , 2013 .

[22]  Meng Joo Er,et al.  Parsimonious Extreme Learning Machine Using Recursive Orthogonal Least Squares , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Mauro Birattari,et al.  Recursive Lazy Learning for Modeling and Control , 1998, ECML.

[24]  Yuan Lan,et al.  Ensemble of online sequential extreme learning machine , 2009, Neurocomputing.

[25]  Chuanhou Gao,et al.  A comparative analysis of support vector machines and extreme learning machines , 2012, Neural Networks.

[26]  Yuan Lan,et al.  Constructive hidden nodes selection of extreme learning machine for regression , 2010, Neurocomputing.

[27]  Zexuan Zhu,et al.  A fast pruned-extreme learning machine for classification problem , 2008, Neurocomputing.

[28]  W. Murray,et al.  An algorithm for RLS identification parameters that vary quickly with time , 1993, IEEE Trans. Autom. Control..

[29]  Qinghua Zheng,et al.  Regularized Extreme Learning Machine , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[30]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[31]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[32]  Wang Hongli.  Time Series Prediction Based on Sequential Regularized Extreme Learning Machine and Its Application , 2011 .

[33]  Yong-Ping Zhao,et al.  Fast cross validation for regularized extreme learning machine , 2014 .

[34]  Li Pinga.  Fast Leave-One-Out Cross-Validation Algorithm for Extreme Learning Machine , 2011 .

[35]  Huijuan Lu,et al.  Text categorization based on regularization extreme learning machine , 2011, Neural Computing and Applications.

[36]  Meng Joo Er,et al.  Constructive multi-output extreme learning machine with application to large tanker motion dynamics identification , 2014, Neurocomputing.

[37]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[38]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[39]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[40]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[41]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.