SVM–ELM: Pruning of Extreme Learning Machine with Support Vector Machines for Regression

Abstract Extreme Learning Machine provides very competitive performance to other related classical predictive models for solving problems such as regression, clustering, and classification. An ELM possesses the advantage of faster computational time in both training and testing. However, one of the main challenges of an ELM is the selection of the optimal number of hidden nodes. This paper presents a new approach to node selection of an ELM based on a 1-norm support vector machine (SVM). In this method, the targets of SVM yi ∈{+1, –1} are derived using the mean or median of ELM training errors as a threshold for separating the training data, which are projected to SVM dimensions. We present an integrated architecture that exploits the sparseness in solution of SVM to prune out the inactive hidden nodes in ELM. Several experiments are conducted on real-world benchmark datasets, and the results attained attest to the efficiency of the proposed method.

[1]  Cheng Wu,et al.  Semi-Supervised and Unsupervised Extreme Learning Machines , 2014, IEEE Transactions on Cybernetics.

[2]  Min Han,et al.  The hidden neurons selection of the wavelet networks using support vector machines and ridge regression , 2008, Neurocomputing.

[3]  Chang Feng,et al.  Meta-ELM: ELM with ELM hidden nodes , 2014, Neurocomputing.

[4]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[5]  Amaury Lendasse,et al.  OP-ELM: Theory, Experiments and a Toolbox , 2008, ICANN.

[6]  Ning Wang,et al.  A Generalized Ellipsoidal Basis Function Based Online Self-constructing Fuzzy Neural Network , 2011, Neural Processing Letters.

[7]  Joachim Diederich,et al.  Rule Extraction from Support Vector Machines , 2008, Studies in Computational Intelligence.

[8]  Hai-Jun Rong,et al.  Aircraft recognition using modular extreme learning machine , 2014, Neurocomputing.

[9]  Yuan Lan,et al.  Constructive hidden nodes selection of extreme learning machine for regression , 2010, Neurocomputing.

[10]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[11]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[12]  César Hervás-Martínez,et al.  PCA-ELM: A Robust and Pruned Extreme Learning Machine Approach Based on Principal Component Analysis , 2012, Neural Processing Letters.

[13]  Meng Joo Er,et al.  Parsimonious Extreme Learning Machine Using Recursive Orthogonal Least Squares , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[14]  C. Jutten,et al.  Gal: Networks That Grow When They Learn and Shrink When They Forget , 1991 .

[15]  Xiang Li,et al.  An Online Self-Organizing Scheme for Parsimonious and Accurate Fuzzy Neural Networks , 2010, Int. J. Neural Syst..

[16]  Olvi L. Mangasarian,et al.  Exact 1-Norm Support Vector Machines Via Unconstrained Convex Differentiable Minimization , 2006, J. Mach. Learn. Res..

[17]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[18]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[19]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[20]  Glenn Fung,et al.  A Feature Selection Newton Method for Support Vector Machine Classification , 2004, Comput. Optim. Appl..

[21]  Jiang Qian,et al.  OMP-ELM: Orthogonal Matching Pursuit-Based Extreme Learning Machine for Regression , 2015, J. Intell. Syst..

[22]  Meng Joo Er,et al.  A fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks , 2009, Neurocomputing.

[23]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[24]  Ethem Alpaydin,et al.  Introduction to machine learning , 2004, Adaptive computation and machine learning.

[25]  Guang-Bin Huang,et al.  Face recognition based on extreme learning machine , 2011, Neurocomputing.

[26]  José Alí Moreno,et al.  Fast Monte Carlo reliability evaluation using support vector machine , 2002, Reliab. Eng. Syst. Saf..

[27]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[28]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[29]  S. Arabia,et al.  Systems of Navier-Stokes equations on Cantor sets , 2013 .

[30]  K. Sunat,et al.  A comparative study of pseudo-inverse computing for the extreme learning machine classifier , 2011, The 3rd International Conference on Data Mining and Intelligent Information Technology Applications.

[31]  Zexuan Zhu,et al.  A fast pruned-extreme learning machine for classification problem , 2008, Neurocomputing.

[32]  Korris Fu-Lai Chung,et al.  Kernel Density Estimation, Kernel Methods, and Fast Learning in Large Data Sets , 2014, IEEE Transactions on Cybernetics.

[33]  K. Gnana Sheela,et al.  Review on Methods to Fix Number of Hidden Neurons in Neural Networks , 2013 .

[34]  Li Zhang,et al.  On the sparseness of 1-norm support vector machines , 2010, Neural Networks.