Self-organization and lateral interaction in echo state network reservoirs

Abstract Echo state networks (ESNs) are recurrent structures that give rise to an interesting trade-off between achievable performance and tractability. This is a consequence of the fact that the key element of these networks – the recurrent intermediate layer known as dynamical reservoir – is not, as a rule, subject to supervised training, which is restricted to the linear output layer, also termed as readout. This trade-off, aside from being of theoretical significance, establishes ESNs as most attractive tools for both online and offline information processing. There are two key aspects to be taken into account in the ESN design: (i) the unsupervised definition of the synaptic weights of the reservoir and (ii) the definition of the structure and of the training strategy associated with the readout. This work is concerned with the first of these aspects: it proposes novel strategies for ESN reservoir design based on the theoretical framework built by Kohonen׳s classical works on self-organization – which includes the notions of short-range positive feedback and lateral inhibition – and also on the related and more recent notion of neural gas. It is shown, with the aid of a representative set of simulation results, that the proposed methodologies are capable of leading to significant performance improvements in the context of relevant information processing tasks – channel equalization and chaotic time series prediction – particularly when the input data suits well a cluster-based profile.

[1]  Roman Bek,et al.  Discourse on one way in which a quantum-mechanics language on the classical logical base can be built up , 1978, Kybernetika.

[2]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[3]  Jochen J. Steil,et al.  Intrinsic plasticity for reservoir learning algorithms , 2007, ESANN.

[4]  Benjamin Schrauwen,et al.  Adapting reservoirs to get Gaussian distributions , 2007 .

[5]  Mantas Lukoševičius,et al.  On self-organizing reservoirs and their hierarchies , 2010 .

[6]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[7]  Hans-Georg Zimmermann,et al.  Recurrent Neural Networks are Universal approximators , 2007, Int. J. Neural Syst..

[8]  Stefan J. Kiebel,et al.  Re-visiting the echo state property , 2012, Neural Networks.

[9]  Minoru Asada,et al.  Initialization and self‐organized optimization of recurrent neural network connectivity , 2009, HFSP journal.

[10]  Jochen Triesch,et al.  A Gradient Rule for the Plasticity of a Neuron's Intrinsic Excitability , 2005, ICANN.

[11]  Gerardo Rubino,et al.  Self-Organizing Maps and Scale-Invariant Maps in Echo State Networks , 2011, 2011 11th International Conference on Intelligent Systems Design and Applications.

[12]  Ricardo Suyama,et al.  Unsupervised Signal Processing , 2010 .

[13]  Fernando José Von Zuben,et al.  Performance analysis of nonlinear echo state network readouts in signal processing tasks , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[14]  José Carlos Príncipe,et al.  Analysis and Design of Echo State Networks , 2007, Neural Computation.

[15]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[16]  Bernd Fritzke,et al.  A Growing Neural Gas Network Learns Topologies , 1994, NIPS.

[17]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[18]  Benjamin Schrauwen,et al.  Memory versus non-linearity in reservoirs , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[19]  Tadashi Yamazaki,et al.  The cerebellum as a liquid state machine , 2007, Neural Networks.

[20]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[21]  C. Malsburg.  Self-organization of orientation sensitive cells in the striate cortex , 2004, Kybernetik.

[22]  S. Strogatz.  Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering , 1995 .

[23]  Jürgen Schmidhuber,et al.  Training Recurrent Networks by Evolino , 2007, Neural Computation.

[24]  Robert M. May,et al.  Simple mathematical models with very complicated dynamics , 1976, Nature.

[25]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[26]  T. Kohonen.  Self-organized formation of topographically correct feature maps , 1982 .

[27]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[28]  H. Kantz,et al.  Nonlinear time series analysis , 1997 .

[29]  Fernando José Von Zuben,et al.  An extended echo state network using Volterra filtering and principal component analysis , 2012, Neural Networks.

[30]  Fernando José Von Zuben,et al.  An echo state network architecture based on volterra filtering and PCA with application to the channel equalization problem , 2011, The 2011 International Joint Conference on Neural Networks.

[31]  John G. Proakis,et al.  Digital Communications , 1983 .

[32]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[33]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[34]  S. Strogatz.  Exploring complex networks , 2001, Nature.

[35]  Gordon Pipa,et al.  SORN: A Self-Organizing Recurrent Neural Network , 2009, Front. Comput. Neurosci..

[36]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[37]  MaassWolfgang,et al.  Real-time computing without stable states , 2002 .

[38]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[39]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[40]  Henry D. I. Abarbanel,et al.  Analysis of Observed Chaotic Data , 1995 .

[41]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[42]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[43]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[44]  D. O. Hebb,et al.  The organization of behavior , 1988 .