Modular state space of echo state network

Echo state network (ESN) mainly consists of a reservoir with a large number of neurons that are randomly connected and a linear readout (output) that is easily adapted. From this point, the reservoir will reconstruct the input signals in the high-dimensional state space. In this paper, modular state space of echo state network (MSSESN) is proposed. First, the state space is divided into several subspaces and each of which is called ''a module''. And then, linear readout of ESN is replaced by piecewise output function which maps each module states individually to the actual output. Furthermore, unlike iterative prediction in ESN, the feedback connections from the output neuron to the reservoir are eliminated, which establishes a direct relationship between the reservoir and output. Finally, the final results can be obtained by assembling the outputs of each module. Different from previous reservoir computing methods, MSSESN takes advantage of the modularity and reservoir mechanisms. It is theoretically analyzed and tested by the benchmark prediction of Mackey-Glass and Lorenz time series. The results have proven the effectiveness of this methodology.

[1]  Babak Nadjar Araabi,et al.  Predicting Chaotic Time Series Using Neural and Neurofuzzy Models: A Comparative Study , 2006, Neural Processing Letters.

[2]  L. Cao.  Practical method for determining the minimum embedding dimension of a scalar time series , 1997 .

[3]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[4]  Herbert Jaeger,et al.  Echo state network , 2007, Scholarpedia.

[5]  Abbas Erfanian Omidvar.  Configuring radial basis function network using fractal scaling process with application to chaotic time series prediction , 2004 .

[6]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[7]  Simon Haykin,et al.  Decoupled echo state networks with lateral inhibition , 2007, Neural Networks.

[8]  Saeed Zolfaghari,et al.  Chaotic time series prediction with residual analysis method using hybrid Elman-NARX neural networks , 2010, Neurocomputing.

[9]  Jürgen Schmidhuber,et al.  Training Recurrent Networks by Evolino , 2007, Neural Computation.

[10]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[11]  Peter Michael Young,et al.  A tighter bound for the echo state property , 2006, IEEE Transactions on Neural Networks.

[12]  Jun Zhang,et al.  Chaotic Time Series Prediction Using a Neuro-Fuzzy System with Time-Delay Coordinates , 2008, IEEE Transactions on Knowledge and Data Engineering.

[13]  Zhidong Deng,et al.  Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution , 2007, IEEE Transactions on Neural Networks.

[14]  Zuren Feng,et al.  Effects of connectivity structure of complex echo state network on its prediction performance for nonlinear time series , 2010, Neurocomputing.

[15]  Min Han,et al.  Prediction of chaotic time series based on the recurrent predictor neural network , 2004, IEEE Transactions on Signal Processing.

[16]  Hubert Cardot,et al.  SOM time series clustering and prediction with recurrent neural networks , 2011, Neurocomputing.

[17]  Herbert Jaeger,et al.  Optimization and applications of echo state networks with leaky- integrator neurons , 2007, Neural Networks.

[18]  Kara M. Kockelman,et al.  Chaos Theory and Transportation Systems: Instructive Example , 2004 .

[19]  Minoru Asada,et al.  Studies on reservoir initialization and dynamics shaping in echo state networks , 2009, ESANN.

[20]  S. Haykin,et al.  Making sense of a complex world [chaotic events modeling] , 1998, IEEE Signal Process. Mag..

[21]  Jiashu Zhang,et al.  Pipelined Chebyshev Functional Link Artificial Recurrent Neural Network for Nonlinear Adaptive Filter , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[22]  Li-yun Su,et al.  Prediction of multivariate chaotic time series with local polynomial fitting , 2010, Comput. Math. Appl..

[23]  Hendrik Van Brussel,et al.  Pruning and regularization in reservoir computing , 2009, Neurocomputing.

[24]  Claudio Gallicchio,et al.  Architectural and Markovian factors of echo state networks , 2011, Neural Networks.

[25]  Yiannis Demiris,et al.  Echo State Gaussian Process , 2011, IEEE Transactions on Neural Networks.

[26]  Helmut Hauser,et al.  Echo state networks with filter neurons and a delay&sum readout , 2010, Neural Networks.

[27]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[28]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[29]  Hossein Mirzaee.  Linear combination rule in genetic algorithm for optimization of finite impulse response neural network to predict natural chaotic time series , 2009 .

[30]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .