Single- and multi-objective particle swarm optimization of reservoir structure in Echo State Network

Echo State Networks ESNs are specific kind of recurrent networks providing a black box modeling of dynamic non-linear problems. Their architecture is distinguished by a randomly recurrent hidden infra-structure called dynamic reservoir. Coming up with an efficient reservoir structure depends mainly on selecting the right parameters including the number of neurons and connectivity rate within it. Despite expertise and repeatedly tests, the optimal reservoir topology is hard to be determined in advance. Topology evolving can provide a potential way to define a suitable reservoir according to the problem to be modeled. This last can be mono- or multi-constrained. Throughout this paper, a mono-objective as well as a multi-objective particle swarm optimizations are applied to ESN to provide a set of optimal reservoir architectures. Both accuracy and complexity of the network are considered as objectives to be optimized during the evolution process. These approaches are tested on various benchmarks such as NARMA and Lorenz time series.

[1]  Vladimir Ceperic,et al.  Reducing Complexity of Echo State Networks with Sparse Linear Regression Algorithms , 2014, 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation.

[2]  Yourong Li,et al.  Short-term fault prediction based on support vector machines with parameter optimization by evolution strategy , 2009, Expert Syst. Appl..

[3]  Xinjie Yu,et al.  Introduction to evolutionary algorithms , 2010, The 40th International Conference on Computers & Indutrial Engineering.

[4]  Adel M. Alimi,et al.  A Hybrid Approach Based on Particle Swarm Optimization for Echo State Network Initialization , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[5]  Voratas Kachitvichyanukul,et al.  Comparison of Three Evolutionary Algorithms: GA, PSO, and DE , 2012 .

[6]  Peter Tiño,et al.  Predictive Modeling with Echo State Networks , 2008, ICANN.

[7]  Adel M. Alimi,et al.  Designing Beta Basis Function Neural Network for optimization using Artificial Bee Colony (ABC) , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[8]  Xin-She Yang,et al.  Metaheuristic Optimization: Algorithm Analysis and Open Problems , 2011, SEA.

[9]  Nikolay I. Nikolaev,et al.  Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling , 2010, IEEE Transactions on Neural Networks.

[10]  B. Malakooti,et al.  On training of artificial neural networks , 1989, International 1989 Joint Conference on Neural Networks.

[11]  Herbert Jaeger,et al.  A tutorial on training recurrent neural networks , covering BPPT , RTRL , EKF and the " echo state network " approach - Semantic Scholar , 2005 .

[12]  Enrique Alba,et al.  An experimental analysis of the Echo State Network initialization using the Particle Swarm Optimization , 2014, 2014 Sixth World Congress on Nature and Biologically Inspired Computing (NaBIC 2014).

[13]  Keith L. Downing,et al.  Introduction to Evolutionary Algorithms , 2006 .

[14]  Yuan Chen,et al.  A new parameters joint optimization method of chaotic time series prediction , 2011 .

[15]  Pierre Roussel-Ragot,et al.  Training recurrent neural networks: why and how? An illustration in dynamical process modeling , 1994, IEEE Trans. Neural Networks.

[16]  André Frank Krause,et al.  Multiobjective optimization of echo state networks for multiple motor pattern learning , 2010 .

[17]  Marco Laumanns,et al.  Why Quality Assessment Of Multiobjective Optimizers Is Difficult , 2002, GECCO.

[18]  Adel M. Alimi,et al.  The Modified Differential Evolution and the RBF (MDE-RBF) Neural Network for Time Series Prediction , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[19]  Geoffrey E. Hinton,et al.  Generating Text with Recurrent Neural Networks , 2011, ICML.

[20]  Oscar Castillo,et al.  Interval type-2 fuzzy weight adjustment for backpropagation neural networks with application in time series prediction , 2014, Inf. Sci..

[21]  Héctor Pomares,et al.  Time series analysis using normalized PG-RBF network with regression weights , 2002, Neurocomputing.

[22]  Teresa Bernarda Ludermir,et al.  Comparing evolutionary methods for reservoir computing pre-training , 2011, The 2011 International Joint Conference on Neural Networks.

[23]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[24]  Les E. Atlas,et al.  Recurrent neural networks and robust time series prediction , 1994, IEEE Trans. Neural Networks.

[25]  C.A. Coello Coello,et al.  MOPSO: a proposal for multiple objective particle swarm optimization , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[26]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[27]  Adel M. Alimi,et al.  Designing of Beta Basis Function Neural Network for optimization using cuckoo search (CS) , 2014, 2014 14th International Conference on Hybrid Intelligent Systems.

[28]  D. N. Tibarewala,et al.  A back-propagation through time based recurrent neural network approach for classification of cognitive EEG states , 2015, 2015 IEEE International Conference on Engineering and Technology (ICETECH).

[29]  Héctor Pomares,et al.  Soft-computing techniques and ARMA model for time series prediction , 2008, Neurocomputing.

[30]  J. Teo,et al.  Multi-Objective vs. Single-Objective Evolutionary Algorithms for hybrid mobile robot optimization , 2014, 2014 IEEE International Symposium on Robotics and Manufacturing Automation (ROMA).

[31]  Adel M. Alimi,et al.  Fuzzy Ant Supervised by PSO and simplified ant supervised PSO applied to TSP , 2013, 13th International Conference on Hybrid Intelligent Systems (HIS 2013).

[32]  Adel M. Alimi,et al.  Opposition-based particle swarm optimization for the design of beta basis function neural network , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[33]  Benjamin Schrauwen,et al.  Reservoir Computing Trends , 2012, KI - Künstliche Intelligenz.