An Improved Echo State Network via L1-Norm Regularization

Considering the ill-posed problem and the model scale control of echo state network, an improved echo state network based on L1-norm regularization is proposed. In order to improve the numerical stability, the proposed method adds an L1-norm penalty term in the objective function. Meanwhile, the method can also control the complexity of the network and prevent overfitting by using feature selection capability of L1-norm regularization. To solve the L1-norm regularization model, we adopt the least angle regression algorithm to calculate regularization path and select suitable model through Bayesian information criterion, which can avoid the estimations of regularization parameter. The model is applied to the time series predictions of both synthetic dataset and practical dataset. The simulation results show the effectiveness and practicality of the proposed method.

[1]  Zexuan Zhu,et al.  A fast pruned-extreme learning machine for classification problem , 2008, Neurocomputing.

[2]  Qionghai Dai,et al.  From Compressed Sensing to Low-rank Matrix Recovery: Theory and Applications , 2013 .

[3]  Han Min.  An Norm 1 Regularization Term ELM Algorithm Based on Surrogate Function and Bayesian Framework , 2011 .

[4]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[5]  Liu Jian,et al.  Classification Algorithm of Support Vector Machine via p-norm Regularization , 2012 .

[6]  Hendrik Van Brussel,et al.  Pruning and regularization in reservoir computing , 2009, Neurocomputing.

[7]  Wei Chen,et al.  Zero-norm Penalized Feature Selection Support Vector Machine: Zero-norm Penalized Feature Selection Support Vector Machine , 2011 .

[8]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.

[9]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[10]  Y. Selen,et al.  Model-order selection: a review of information criterion rules , 2004, IEEE Signal Processing Magazine.

[11]  Qiao Jun,et al.  Application of ESN-based Multi Indices Dual Heuristic Dynamic Programming on Wastewater Treatment Process , 2013 .

[12]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[13]  Kang Li,et al.  Variable selection via RIVAL (removing irrelevant variables amidst Lasso iterations) and its application to nuclear material detection , 2012, Autom..

[14]  Sumio Watanabe,et al.  A widely applicable Bayesian information criterion , 2012, J. Mach. Learn. Res..

[15]  Peng Xi-yuan,et al.  Researches on Time Series Prediction with Echo State Networks , 2010 .

[16]  J. Friedman.  Fast sparse regression and classification , 2012 .

[17]  Weiping Zhang,et al.  Control of discrete chaotic systems based on echo state network modeling with an adaptive noise canceler , 2012, Knowl. Based Syst..