Smoothing Regularized Extreme Learning Machine

Extreme learning machines have been applied successfully to many real-world applications, due to their faster training speed and good performance. However, in order to guarantee the convergence of the ELM algorithm, it initially requires a large number of hidden nodes. In addition, extreme learning machines have two drawbacks: over-fitting and the sensitivity of accuracy to the number of hidden nodes. The aim of this paper is to propose a new smoothing \(L_{1/2}\) extreme learning machine with regularization to overcome these two drawbacks. The main advantage of the proposed approach is to reduce weights to smaller values during the training, and such nodes with sufficiently small weights can eventually be removed after training so as to obtain a suitable network size. Numerical experiments have been carried out for approximation problems and multi-class classification problems, and preliminary results have shown that the proposed approach works well.

[1]  Huanhe Dong,et al.  Discrete Symmetries Analysis and Exact Solutions of the Inviscid Burgers Equation , 2012 .

[2]  Jacek M. Zurada,et al.  Batch gradient method with smoothing L1/2 regularization for training of feedforward neural networks , 2014, Neural Networks.

[3]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[4]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[5]  Wang Yao,et al.  L 1/2 regularization , 2010 .

[6]  Wei Wu,et al.  A Smoothing Interval Neural Network , 2012 .

[7]  Xiang-Sun Zhang,et al.  Neural networks in optimization , 2000 .

[8]  Zongben Xu,et al.  Sparse SAR imaging based on L1/2 regularization , 2012, Science China Information Sciences.

[9]  Guang-Bin Huang,et al.  Classification ability of single hidden layer feedforward neural networks , 2000, IEEE Trans. Neural Networks Learn. Syst..

[10]  Tomaso A. Poggio,et al.  Extensions of a Theory of Networks for Approximation and Learning , 1990, NIPS.

[11]  Jacek M. Zurada,et al.  Convergence of online gradient method for feedforward neural networks with smoothing L1/2 regularization penalty , 2014, Neurocomputing.

[12]  George D. Magoulas,et al.  Improving the Convergence of the Backpropagation Algorithm Using Learning Rate Adaptation Methods , 1999, Neural Computation.

[13]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[14]  Jacek M. Zurada,et al.  Competitive Layer Model of Discrete-Time Recurrent Neural Networks with LT Neurons , 2010, Neural Computation.

[15]  Yuhong Dai,et al.  Minimization Algorithms Based on Supervisor and Searcher Cooperation , 2001 .

[16]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[17]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[18]  Li Zhang,et al.  On the sparseness of 1-norm support vector machines , 2010, Neural Networks.

[19]  S. Balasundaram,et al.  1-Norm extreme learning machine for regression and multiclass classification using Newton method , 2014, Neurocomputing.

[20]  K. S. Banerjee.  Generalized Inverse of Matrices and Its Applications , 1973 .

[21]  Amaury Lendasse,et al.  TROP-ELM: A double-regularized ELM using LARS and Tikhonov regularization , 2011, Neurocomputing.

[22]  R. Chartrand,et al.  Restricted isometry properties and nonconvex compressive sensing , 2007 .

[23]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[24]  Chi-Man Vong,et al.  Sparse Bayesian Extreme Learning Machine for Multi-classification , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[26]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[27]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[28]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[29]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[30]  Zongben Xu,et al.  $L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast Solver , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[31]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[32]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[33]  Deyu Meng,et al.  Improve robustness of sparse PCA by L1-norm maximization , 2012, Pattern Recognit..

[34]  Qinghua Zheng,et al.  Regularized Extreme Learning Machine , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.