PSO-based growing echo state network

Abstract Reservoir computing (RC), with the idea of using a large randomly and sparsely connected recurrent layer, has turned out to be an efficient paradigm for training recurrent neural networks. Echo state network (ESN) is one of typical representatives of RC, whose good performance mainly depends on the selection of right parameters. In this paper, we introduce an approach to pre-train a growing ESN with multiple sub-reservoirs by optimizing singular values, based on particle swarm optimization (PSO) and singular value decomposition (SVD). During the network growing, the sub-reservoirs are added to the network one by one, which leads to a quasi-diagonal reservoir weight matrix. For each sub-reservoir, before being added to the network, the singular values are optimized by PSO, then the sub-reservoir weight matrix is designed by SVD with the optimized singular values. In order to evaluate the performance of sub-reservoirs during optimization, a novel fitness function is derived based on some concepts in vector space. Finally, simulation results, on both benchmark and real-world data sets, show the superior performance of the proposed method.

[1]  Adel M. Alimi,et al.  PSO-based analysis of Echo State Network parameters for time series forecasting , 2017, Appl. Soft Comput..

[2]  Christian Igel,et al.  Structure optimization of reservoir networks , 2010, Log. J. IGPL.

[3]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[4]  Roger Labahn,et al.  Design Strategies for Weight Matrices of Echo State Networks , 2012, Neural Computation.

[5]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[6]  Jiabin Wang,et al.  Functional echo state network for time series classification , 2016, Inf. Sci..

[7]  Witold Pedrycz,et al.  Hybrid Neural Prediction and Optimized Adjustment for Coke Oven Gas System in Steel Industry , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[9]  Akinori Nishihara,et al.  Evolutionary pre-training for CRJ-type reservoir of echo state networks , 2015, Neurocomputing.

[10]  Junfei Qiao,et al.  Growing Echo-State Network With Multiple Subreservoirs , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[11]  Eduardo Sontag,et al.  Turing computability with neural nets , 1991 .

[12]  Yiannis Demiris,et al.  Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[14]  Xuefeng Yan,et al.  Optimizing the echo state network based on mutual information for modeling fed-batch bioprocesses , 2017, Neurocomputing.

[15]  Mantas Lukosevicius,et al.  A Practical Guide to Applying Echo State Networks , 2012, Neural Networks: Tricks of the Trade.

[16]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[17]  Zhidong Deng,et al.  Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution , 2007, IEEE Transactions on Neural Networks.

[18]  Yuhui Shi,et al.  Particle swarm optimization: developments, applications and resources , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[19]  Simon Haykin,et al.  Decoupled echo state networks with lateral inhibition , 2007, Neural Networks.

[20]  Lei Wang,et al.  Design of polynomial echo state networks for time series prediction , 2018, Neurocomputing.

[21]  Amir Hussain,et al.  Multilayered Echo State Machine: A Novel Architecture and Algorithm , 2017, IEEE Transactions on Cybernetics.

[22]  Xiaohua Wang,et al.  Echo State Networks With Orthogonal Pigeon-Inspired Optimization for Image Restoration , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Danilo P. Mandic,et al.  An Augmented Echo State Network for Nonlinear Adaptive Filtering of Complex Noncircular Signals , 2011, IEEE Transactions on Neural Networks.

[24]  Yan Meng,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[25]  Lin Lin,et al.  Genetic algorithm optimized double-reservoir echo state network for multi-regime time series prediction , 2017, Neurocomputing.

[26]  Danilo P. Mandic,et al.  Quaternion-Valued Echo State Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Henry Leung,et al.  Prediction Intervals for a Noisy Nonlinear Time Series Based on a Bootstrapping Reservoir Computing Network Ensemble , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Xuefeng Yan,et al.  Optimizing the echo state network with a binary particle swarm optimization algorithm , 2015, Knowl. Based Syst..

[29]  Hans-Georg Zimmermann,et al.  Recurrent Neural Networks are Universal approximators , 2007, Int. J. Neural Syst..

[30]  Zhigang Zeng,et al.  A novel echo state network for multivariate and nonlinear time series prediction , 2018, Appl. Soft Comput..

[31]  John G. Harris,et al.  Automatic speech recognition using a predictive echo state network classifier , 2007, Neural Networks.

[32]  Pericles A. Mitkas,et al.  Adaptive reservoir computing through evolution and learning , 2013, Neurocomputing.

[33]  Xue-feng Yan,et al.  Reservoir Computing with Sensitivity Analysis Input Scaling Regulation and Redundant Unit Pruning for Modeling Fed-Batch Bioprocesses , 2014 .

[34]  Lorenzo Livi,et al.  Investigating echo state networks dynamics by means of recurrence analysis , 2016, IEEE Trans. Neural Networks Learn. Syst..

[35]  Marc Schoenauer,et al.  Supervised and Evolutionary Learning of Echo State Networks , 2008, PPSN.

[36]  Sadiq M. Sait,et al.  Applications of Metaheuristics in Reservoir Computing Techniques: A Review , 2018, IEEE Access.

[37]  Min Han,et al.  Laplacian Echo State Network for Multivariate Time Series Prediction , 2018, IEEE Transactions on Neural Networks and Learning Systems.