A modified extreme learning machine with sigmoidal activation functions

This paper proposes a modified ELM algorithm that properly selects the input weights and biases before training the output weights of single-hidden layer feedforward neural networks with sigmoidal activation function and proves mathematically the hidden layer output matrix maintains full column rank. The modified ELM avoids the randomness compared with the ELM. The experimental results of both regression and classification problems show good performance of the modified ELM algorithm.

[1]  Tony R. Martinez,et al.  Heterogeneous radial basis function networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[2]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[3]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[4]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[5]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[6]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[7]  E. Romero,et al.  A new incremental method for function approximation using feed-forward neural networks , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[8]  Zongben Xu,et al.  Simultaneous Lp-approximation order for neural networks , 2005, Neural Networks.

[9]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[10]  K. S. Banerjee.  Generalized Inverse of Matrices and Its Applications , 1973 .

[11]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[12]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[13]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[14]  Gunnar Rätsch,et al.  An Improvement of AdaBoost to Avoid Overfitting , 1998, ICONIP.

[15]  Zongben Xu,et al.  Approximation capability of interpolation neural networks , 2010, Neurocomputing.

[16]  Tianping Chen,et al.  Approximation capability to functions of several variables, nonlinear functionals and operators by radial basis function neural networks , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[17]  C. R. Rao,et al.  Generalized Inverse of Matrices and its Applications , 1972 .

[18]  Feilong Cao,et al.  A study on effectiveness of extreme learning machine , 2011, Neurocomputing.

[19]  Rui Zhang,et al.  The errors of approximation for feedforward neural networks in the Lp metric , 2009, Math. Comput. Model..

[20]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[21]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[22]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[23]  Yuan Lan,et al.  Random search enhancement of error minimized extreme learning machine , 2010, ESANN.

[24]  XUZongben,et al.  The essential order of approximation for neural networks , 2004 .

[25]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[26]  Bum Il Hong,et al.  An approximation by neural networkswith a fixed weight , 2004 .

[27]  Hong Chen,et al.  Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems , 1995, IEEE Trans. Neural Networks.

[28]  D. Serre.  Matrices: Theory and Applications , 2002 .

[29]  Feilong Cao,et al.  Interpolation and rates of convergence for a class of neural networks , 2009 .

[30]  Zongben Xu,et al.  The estimate for approximation error of neural networks: A constructive approach , 2008, Neurocomputing.

[31]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.