A sparse extreme learning machine framework by continuous optimization algorithms and its application in pattern recognition

Extreme learning machine (ELM) has demonstrated great potential in machine learning owing to its simplicity, rapidity and good generalization performance. In this investigation, based on least-squares estimate (LSE) and least absolute deviation (LAD), we propose four sparse ELM formulations with zero-norm regularization to automatically choose the optimal hidden nodes. Furthermore, we develop two continuous optimization methods to solve the proposed problems respectively. The first is DC (difference of convex functions) approximation approach that approximates the zero-norm by a DC function, and the resulting optimizations are posed as DC programs. The second is an exact penalty technique for zero-norm, and the resulting problems are reformulated as DC programs, and the corresponding DCAs converge finitely. Moreover, the proposed framework is applied directly to recognize the hardness of licorice seeds using near-infrared spectral data. Experiments in different spectral regions illustrate that the proposed approaches can reduce the number of hidden nodes (or output features), while either improve or show no significant difference in generalization compared with the traditional ELM methods and support vector machine (SVM). Experiments on several benchmark data sets demonstrate that the proposed framework is competitive with the traditional approaches in generalization, but selects fewer output features. A sparse ELM framework is proposed based on with zero-norm regularization.Four sparse ELM formulations with zero-norm are built based on LSE and LAD.We develop two continuous approaches to solve the problems.The first is DC (difference of convex functions) approximation approach.The second is an exact penalty technique for zero-norm.All the resulting problems are posed as DC programming.

[1]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[2]  Amaury Lendasse,et al.  Regularized extreme learning machine for regression with missing data , 2013, Neurocomputing.

[3]  Yiguang Liu,et al.  Estimating the fundamental matrix based on least absolute deviation , 2011, Neurocomputing.

[4]  S. Balasundaram,et al.  1-Norm extreme learning machine for regression and multiclass classification using Newton method , 2014, Neurocomputing.

[5]  Liming Yang,et al.  Recognition of the hardness of licorice seeds using a semi-supervised learning method and near-infrared spectral data , 2012 .

[6]  Cao Hui-rong,et al.  System Identification Based on the Least-Absolute Criteria , 2009 .

[7]  Wei Jiang,et al.  Fast sparse approximation of extreme learning machine , 2014, Neurocomputing.

[8]  Alexandros Iosifidis,et al.  Regularized Extreme Learning Machine for Large-scale Media Content Analysis , 2015, INNS Conference on Big Data.

[9]  Zhengyan Lin,et al.  Adaptive Lasso in high-dimensional settings , 2009 .

[10]  Le Thi Hoai An,et al.  DC approximation approaches for sparse optimization , 2014, Eur. J. Oper. Res..

[11]  Le Thi Hoai An,et al.  Feature selection in machine learning: an exact penalty approach using a Difference of Convex function Algorithm , 2014, Machine Learning.

[12]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[13]  김용수,et al.  Extreme Learning Machine 기반 퍼지 패턴 분류기 설계 , 2015 .

[14]  Feiping Nie,et al.  Discriminative Least Squares Regression for Multiclass Classification and Feature Selection , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Punyaphol Horata,et al.  Robust extreme learning machine , 2013, Neurocomputing.

[16]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.

[17]  Yuehua Wu,et al.  General M-estimation , 1997 .

[18]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[19]  Guang-Bin Huang,et al.  Trends in extreme learning machines: A review , 2015, Neural Networks.

[20]  Hongming Zhou,et al.  Optimization method based extreme learning machine for classification , 2010, Neurocomputing.

[21]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[22]  Guilherme De A. Barreto,et al.  A Robust Extreme Learning Machine for pattern classification with outliers , 2016, Neurocomputing.

[23]  Wei Wu,et al.  Double parallel feedforward neural network based on extreme learning machine with L1/2 regularizer , 2014, Neurocomputing.

[24]  Z. D. Bai,et al.  GeneralM-Estimation , 1997 .

[25]  H. Mishra,et al.  Fourier Transform Near-Infrared Spectroscopy for rapid and simple determination of phytic acid content in green gram seeds (Vigna radiata). , 2015, Food chemistry.

[26]  Benoît Frénay,et al.  Feature selection for nonlinear models with extreme learning machines , 2013, Neurocomputing.

[27]  Chuanhou Gao,et al.  A comparative analysis of support vector machines and extreme learning machines , 2012, Neural Networks.

[28]  Amaury Lendasse,et al.  TROP-ELM: A double-regularized ELM using LARS and Tikhonov regularization , 2011, Neurocomputing.

[29]  Amaury Lendasse,et al.  HSR: L1/2-regularized sparse representation for fast face recognition using hierarchical feature selection , 2014, Neural Computing and Applications.

[30]  Minxia Luo,et al.  A hybrid approach combining extreme learning machine and sparse representation for image classification , 2014, Eng. Appl. Artif. Intell..

[31]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[32]  Le Thi Hoai An,et al.  A DC programming approach for feature selection in support vector machines learning , 2008, Adv. Data Anal. Classif..