Optimization method based extreme learning machine for classification

Extreme learning machine (ELM) as an emergent technology has shown its good performance in regression applications as well as in large dataset (and/or multi-label) classification applications. The ELM theory shows that the hidden nodes of the ''generalized'' single-hidden layer feedforward networks (SLFNs), which need not be neuron alike, can be randomly generated and the universal approximation capability of such SLFNs can be guaranteed. This paper further studies ELM for classification in the aspect of the standard optimization method and extends ELM to a specific type of ''generalized'' SLFNs-support vector network. This paper shows that: (1) under the ELM learning framework, SVM's maximal margin property and the minimal norm of weights theory of feedforward neural networks are actually consistent; (2) from the standard optimization method point of view ELM for classification and SVM are equivalent but ELM has less optimization constraints due to its special separability feature; (3) as analyzed in theory and further verified by the simulation results, ELM for classification tends to achieve better generalization performance than traditional SVM. ELM for classification is less sensitive to user specified parameters and can be implemented easily.

[1]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[2]  Chris H. Q. Ding,et al.  Minimum redundancy feature selection from microarray gene expression data , 2003, Computational Systems Bioinformatics. CSB2003. Proceedings of the 2003 IEEE Bioinformatics Conference. CSB2003.

[3]  Narasimhan Sundararajan,et al.  Online Sequential Fuzzy Extreme Learning Machine for Function Approximation and Classification Problems , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[4]  J. Mesirov,et al.  Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. , 1999, Science.

[5]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[6]  Frank Rosenblatt,et al.  PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS , 1963 .

[7]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Guang-Bin Huang,et al.  Convex Incremental Extreme Learning Machine , 2007 .

[9]  T. M. Williams.  Practical Methods of Optimization. Vol. 2 — Constrained Optimization , 1982 .

[10]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[11]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[12]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[13]  Qing He,et al.  Extreme Support Vector Machine Classifier , 2008, PAKDD.

[14]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[15]  Chee Kheong Siew,et al.  Can threshold networks be trained directly? , 2006, IEEE Transactions on Circuits and Systems II: Express Briefs.

[16]  Glenn Fung,et al.  Proximal support vector machine classifiers , 2001, KDD '01.

[17]  Nikhil R. Pal,et al.  NEUROSVM: An Architecture to Reduce the Effect of the Choice of Kernel on the Performance of SVM , 2009, J. Mach. Learn. Res..

[18]  Narasimhan Sundararajan,et al.  Fully complex extreme learning machine , 2005, Neurocomputing.

[19]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[20]  Benoît Frénay,et al.  Using SVMs with randomised feature spaces: an extreme learning approach , 2010, ESANN.

[21]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[22]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[23]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[24]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[25]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.