Low complexity adaptive forgetting factor for online sequential extreme learning machine (OS-ELM) for application to nonstationary system estimations

Huang et al. (2004) has recently proposed an on-line sequential ELM (OS-ELM) that enables the extreme learning machine (ELM) to train data one-by-one as well as chunk-by-chunk. OS-ELM is based on recursive least squares-type algorithm that uses a constant forgetting factor. In OS-ELM, the parameters of the hidden nodes are randomly selected and the output weights are determined based on the sequentially arriving data. However, OS-ELM using a constant forgetting factor cannot provide satisfactory performance in time-varying or nonstationary environments. Therefore, we propose an algorithm for the OS-ELM with an adaptive forgetting factor that maintains good performance in time-varying or nonstationary environments. The proposed algorithm has the following advantages: (1) the proposed adaptive forgetting factor requires minimal additional complexity of O(N) where N is the number of hidden neurons, and (2) the proposed algorithm with the adaptive forgetting factor is comparable with the conventional OS-ELM with an optimal forgetting factor.

[1]  P. Holland,et al.  Robust regression using iteratively reweighted least-squares , 1977 .

[2]  Chee Kheong Siew,et al.  Incremental extreme learning machine with fully complex hidden nodes , 2008, Neurocomputing.

[3]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  The Brazilian Symposium on Neural Networks (SBRN'04) , 2006, Neurocomputing.

[4]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[5]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[6]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[7]  Sangwook Lee,et al.  Recursive Complex Extreme Learning Machine with Widely Linear Processing for Nonlinear Channel Equalizer , 2006, ISNN.

[8]  Robert F. Stengel,et al.  Smooth function approximation using neural networks , 2005, IEEE Transactions on Neural Networks.

[9]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[10]  Eric B. Baum,et al.  On the capabilities of multilayer perceptrons , 1988, J. Complex..

[11]  Koeng-Mo Sung,et al.  Robust Kalman Filtering with Variable Forgetting Factor against Impulsive Noise , 2001 .

[12]  Simon Haykin,et al.  Adaptive Filter Theory 4th Edition , 2002 .

[13]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[14]  Jacob Benesty,et al.  A Robust Variable Forgetting Factor Recursive Least-Squares Algorithm for System Identification , 2008, IEEE Signal Processing Letters.

[15]  Bin Yang,et al.  Projection approximation subspace tracking , 1995, IEEE Trans. Signal Process..

[16]  Theodore S. Rappaport,et al.  Wireless communications - principles and practice , 1996 .

[17]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[18]  Sang-Wook Lee,et al.  Channel estimation for OFDM with fast fading channels by modified Kalman filter , 2004, IEEE Trans. Consumer Electron..

[19]  Maciej Niedzwiecki,et al.  Identification of Time-Varying Processes , 2000 .

[20]  Yoh-Han Pao,et al.  Stochastic choice of basis functions in adaptive function approximation and the functional-link net , 1995, IEEE Trans. Neural Networks.

[21]  Sou-Chen Lee,et al.  An efficient on-line thermal input estimation method using kalman filter and recursive least square algorithm , 1997 .

[22]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[23]  Koeng-Mo Sung,et al.  Reduced Complexity Self-Tuning Adaptive Algorithms in Application to Channel Estimation , 2007, IEEE Transactions on Communications.

[24]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[25]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[26]  Seokjin Lee,et al.  A low-complexity AFF-RLS algorithm using a normalization technique , 2009, IEICE Electron. Express.

[27]  Woon-Seng Gan,et al.  Subband Adaptive Filtering: Theory and Implementation , 2009 .

[28]  Chee Kheong Siew,et al.  Can threshold networks be trained directly? , 2006, IEEE Transactions on Circuits and Systems II: Express Briefs.

[29]  Xiao Liu,et al.  Canonical piecewise linear network for nonlinear filtering and its application to blind equalization , 1997, Signal Process..

[30]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[31]  D. Lowe,et al.  Adaptive radial basis function nonlinearities, and the problem of generalisation , 1989 .