Incremental and Decremental Extreme Learning Machine Based on Generalized Inverse

In online sequential applications, a machine learning model needs to have a self-updating ability to handle the situation, which the training set is changing. Conventional incremental extreme learning machine (ELM) and online sequential ELM are usually achieved in two approaches: directly updating the output weight and recursively computing the left pseudo inverse of the hidden layer output matrix. In this paper, we develop a novel solution for incremental and decremental ELM (DELM), via recursively updating and downdating the generalized inverse of the hidden layer output matrix. By preserving the global optimality and best generalization performance, our approach implements node incremental ELM (N-IELM) and sample incremental ELM (S-IELM) in a universal form, and overcomes the problem of self-starting and numerical instability in the conventional online sequential ELM. We also propose sample DELM (S-DELM), which is the first decremental version of ELM. The experiments on regression and classification problems with real-world data sets demonstrate the feasibility and effectiveness of the proposed algorithms with encouraging performances.

[1]  Ralph R. Martin,et al.  Merging and Splitting Eigenspace Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[3]  Qingshan Liu,et al.  A Self-Paced Regularization Framework for Multilabel Learning , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[5]  Changsheng Li,et al.  Self-Paced Multi-Task Learning , 2016, AAAI.

[6]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[7]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[8]  Long Lan,et al.  Online Multi-Object Tracking by Quadratic Pseudo-Boolean Optimization , 2016, IJCAI.

[9]  Gene H. Golub,et al.  Matrix computations , 1983 .

[10]  Weihui Dai,et al.  Incremental regularized extreme learning machine and it's enhancement , 2016, Neurocomputing.

[11]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[12]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[13]  Charles A. Micchelli,et al.  How to Choose an Activation Function , 1993, NIPS.

[14]  Felipe Albertao,et al.  Incremental dictionary learning for fault detection with applications to oil pipeline leakage detection , 2011 .

[15]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[16]  Ichiro Takeuchi,et al.  Multiple Incremental Decremental Learning of Support Vector Machines , 2009, IEEE Transactions on Neural Networks.

[17]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[18]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[19]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[20]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[21]  R. Penrose.  A Generalized inverse for matrices , 1955 .

[22]  P. Saratchandran,et al.  Multicategory Classification Using An Extreme Learning Machine for Microarray Gene Expression Cancer Diagnosis , 2007, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[23]  Junchi Yan,et al.  Self-Paced MultiTask Learning , 2017 .

[24]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[25]  Shaoning Pang,et al.  Incremental and decremental LDA learning with applications , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[26]  Adi Ben-Israel,et al.  Generalized inverses: theory and applications , 1974 .

[27]  Cheng-Hao Tsai,et al.  Incremental and decremental training for linear classification , 2014, KDD.

[28]  Jacobson’s lemma for the generalized Drazin inverse , 2012 .

[29]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[30]  W. Marsden.  I and J , 2012 .

[31]  R. E. Cline.  Representations for the Generalized Inverse of a Partitioned Matrix , 1964 .

[32]  Guang-Bin Huang,et al.  Face recognition based on extreme learning machine , 2011, Neurocomputing.

[33]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[34]  Chenping Hou,et al.  One-Pass Learning with Incremental and Decremental Features , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[36]  E. Yaz.  Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.

[37]  Zhongliang Jing,et al.  EVD Dualdating Based Online Subspace Learning , 2014 .

[38]  Yuan Lan,et al.  Random search enhancement of error minimized extreme learning machine , 2010, ESANN.

[39]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .