Balanced echo state networks

This paper investigates the interaction between the driving output feedback and the internal reservoir dynamics in echo state networks (ESNs). The interplay is studied experimentally on the multiple superimposed oscillators (MSOs) benchmark. The experimental data reveals a dual effect of the output feedback strength on the network dynamics: it drives the dynamic reservoir but it can also block suitable reservoir dynamics. Moreover, the data shows that the reservoir size crucially co-determines the likelihood of generating an effective ESN. We show that dependent on the complexity of the MSO dynamics somewhat smaller networks can yield better performance. Optimizing the output feedback weight range and the network size is thus crucial for generating an effective ESN. With proper parameter choices, we show that it is possible to generate ESNs that approximate MSOs with several orders of magnitude smaller errors than those previously reported. We conclude that there appears to be still much more potential in ESNs than previously thought and sketch-out some promising future research directions.

[1]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[2]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[3]  Simon Haykin,et al.  Decoupled echo state networks with lateral inhibition , 2007, Neural Networks.

[4]  Peter Tiño,et al.  Predictive Modeling with Echo State Networks , 2008, ICANN.

[5]  Jochen J. Steil,et al.  Recurrence Enhances the Spatial Encoding of Static Inputs in Reservoir Networks , 2010, ICANN.

[6]  Christian Igel,et al.  Structure optimization of reservoir networks , 2010, Log. J. IGPL.

[7]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[8]  Jürgen Schmidhuber,et al.  Training Recurrent Networks by Evolino , 2007, Neural Computation.

[9]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[10]  Herbert Jaeger,et al.  Optimization and applications of echo state networks with leaky- integrator neurons , 2007, Neural Networks.

[11]  Herbert Jaeger,et al.  A tutorial on training recurrent neural networks , covering BPPT , RTRL , EKF and the " echo state network " approach - Semantic Scholar , 2005 .

[12]  Helmut Hauser,et al.  Echo state networks with filter neurons and a delay&sum readout , 2010, Neural Networks.

[13]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.