An online sequential learning algorithm for regularized Extreme Learning Machine

The Extreme Learning Machine (ELM) is a novel learning scheme for single hidden layer feedforward neural networks, and it has attracted a great deal of research attention since the last decade because of its extremely fast learning speed. One popular variant of ELM is the Online Sequential ELM (OS-ELM), which can deal with sequential learning tasks. However, limitations exist in the OS-ELM such as requiring the initialization phase, pre-defined important parameters, running into singularity problem, inconsistent and potentially unreliable performance. In this paper, an Online Sequential Regularized ELM (OS-RELM) is proposed to address the aforementioned issues. The main idea is to incorporate the regularization method to further improve its generalization performance, and a new update formula is used to eliminate the initialization phase. To enable the OS-RELM to adapt to new data in an effective and reliable manner, an efficient Leave-One-Out Cross-Validation method is implemented. Finally, a matrix reconstruction method is employed to address the unstable update issue. Unlike some ELM variants that greatly jeopardize the speed advantage of the ELM, we strive to limit the computational load from the proposed scheme. Simulation results on benchmark problems show that the OS-RELM is a reliable and efficient algorithm with superior generalization performance than the OS-ELM.

[1]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[2]  Klaus Neumann,et al.  Optimizing extreme learning machines via ridge regression and batch intrinsic plasticity , 2013, Neurocomputing.

[3]  A. Kai Qin,et al.  Evolutionary extreme learning machine , 2005, Pattern Recognit..

[4]  R. H. Myers.  Classical and modern regression with applications , 1986 .

[5]  S. R. Searle,et al.  On Deriving the Inverse of a Sum of Matrices , 1981 .

[6]  Yuan Lan,et al.  Two-stage extreme learning machine for regression , 2010, Neurocomputing.

[7]  Erkki Oja,et al.  GPU-accelerated and parallelized ELM ensembles for large-scale regression , 2011, Neurocomputing.

[8]  Yuan Lan,et al.  Ensemble of online sequential extreme learning machine , 2009, Neurocomputing.

[9]  Yi Liu,et al.  A Decreased Extreme Learning Machine with Ridge Parameter for Online Identification of Nonlinear Systems , 2013 .

[10]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[11]  Guang-Bin Huang,et al.  What are Extreme Learning Machines? Filling the Gap Between Frank Rosenblatt’s Dream and John von Neumann’s Puzzle , 2015, Cognitive Computation.

[12]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[13]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[14]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[15]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[16]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Gavin C. Cawley,et al.  Fast exact leave-one-out cross-validation of sparse least-squares support vector machines , 2004, Neural Networks.

[18]  Zhi-Zhong Mao,et al.  An Ensemble ELM Based on Modified AdaBoost.RT Algorithm for Predicting the Temperature of Molten Steel in Ladle Furnace , 2010, IEEE Transactions on Automation Science and Engineering.

[19]  Visakan Kadirkamanathan,et al.  A Function Estimation Approach to Sequential Learning with Neural Networks , 1993, Neural Computation.

[20]  Qinghua Zheng,et al.  Reduced Kernel Extreme Learning Machine , 2013, CORES.

[21]  Gerhard Tutz,et al.  Boosting ridge regression , 2007, Comput. Stat. Data Anal..

[22]  Narasimhan Sundararajan,et al.  Online Sequential Fuzzy Extreme Learning Machine for Function Approximation and Classification Problems , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Meng Joo Er,et al.  A Study on the Randomness Reduction Effect of Extreme Learning Machine with Ridge Regression , 2013, ISNN.

[24]  Kar-Ann Toh,et al.  Deterministic Neural Classification , 2008, Neural Computation.

[25]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[26]  Qinyu. Zhu.  Extreme Learning Machine , 2013 .

[27]  Chee Kheong Siew,et al.  Incremental extreme learning machine with fully complex hidden nodes , 2008, Neurocomputing.

[28]  Meng Joo Er,et al.  An Enhanced Online Sequential Extreme Learning Machine algorithm , 2008, 2008 Chinese Control and Decision Conference.

[29]  Peter L. Bartlett,et al.  For Valid Generalization the Size of the Weights is More Important than the Size of the Network , 1996, NIPS.

[30]  Meng Joo Er,et al.  Parsimonious Extreme Learning Machine Using Recursive Orthogonal Least Squares , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Wang Xin,et al.  Boosting ridge extreme learning machine , 2012, 2012 IEEE Symposium on Robotics and Applications (ISRA).

[32]  Narasimhan Sundararajan,et al.  An efficient sequential learning algorithm for growing and pruning RBF (GAP-RBF) networks , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[33]  Y Lu,et al.  A Sequential Learning Scheme for Function Approximation Using Minimal Radial Basis Function Neural Networks , 1997, Neural Computation.

[34]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[35]  A.H. Nizar,et al.  Power Utility Nontechnical Loss Analysis With Extreme Learning Machine Method , 2008, IEEE Transactions on Power Systems.

[36]  Guang-Bin Huang,et al.  Learning to Rank with Extreme Learning Machine , 2013, Neural Processing Letters.

[37]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[38]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[39]  Huijuan Lu,et al.  Text categorization based on regularization extreme learning machine , 2011, Neural Computing and Applications.

[40]  Meng Joo Er,et al.  Constructive multi-output extreme learning machine with application to large tanker motion dynamics identification , 2014, Neurocomputing.

[41]  Guang-Bin Huang,et al.  An Insight into Extreme Learning Machines: Random Neurons, Random Features and Kernels , 2014, Cognitive Computation.

[42]  L. BartlettP..  The sample complexity of pattern classification with neural networks , 2006 .

[43]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[44]  Han Wang,et al.  Ensemble Based Extreme Learning Machine , 2010, IEEE Signal Processing Letters.

[45]  Tang Ao,et al.  Batch-to-Batch Iterative Learning Control of a Batch Polymerization Process Based on Online Sequential Extreme Learning Machine , 2009 .