Analysis and Design of Echo State Networks for Function Approximation

The present design of echo state network (ESN) parameters relies on the selection of the maximum eigenvalue of the linearized system around zero. However, this has been found far from optimal for function approximation. This letter presents a function approximation perspective to better understand the operation of ESNs and proposes an information-theoretic measure, the average entropy of echo states, to assess the “richness” of the ESN dynamics. Furthermore, it provides a new interpretation of the ESN dynamics rooted in system theory as a combination of linearized systems where their poles move according to the input signal dynamics. With this interpretation, we will be able to a priori design ESNs with uniform pole distributions covering the frequency spectrum optimally. With adaptive read-outs, the designed ESN can be used as a general infrastructure to represent information in time. 1Correspondence address: Mustafa C. Ozturk, Computational NeuroEngineering Laboratory, Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, Tel: (352)392-2682, Fax: (352)392-0044

[1]  W. Kautz.  Transient synthesis in the time domain , 1954 .

[2]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[3]  S. Liberty,et al.  Linear Systems , 2010, Scientific Parallel Computing.

[4]  Sharad Singhal,et al.  Training Multilayer Perceptrons with the Extende Kalman Algorithm , 1988, NIPS.

[5]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[6]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[7]  Christopher G. Langton,et al.  Computation at the edge of chaos: Phase transitions and emergent computation , 1990 .

[8]  Eduardo Sontag,et al.  Turing computability with neural nets , 1991 .

[9]  Jose C. Principe,et al.  Temporal processing with neural networks: the development of the gamma model , 1991 .

[10]  H. Siegelmann.  Foundations of recurrent neural networks , 1993 .

[11]  José Carlos Príncipe,et al.  The gamma-filter-a new class of adaptive IIR filters with restricted feedback , 1993, IEEE Trans. Signal Process..

[12]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[13]  Lee A. Feldkamp,et al.  Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks , 1994, IEEE Trans. Neural Networks.

[14]  Elias S. Manolakos,et al.  Using recurrent neural networks for adaptive communication channel equalization , 1994, IEEE Trans. Neural Networks.

[15]  Sang-Hoon Oh,et al.  Effect of nonlinear transformations on correlation between weighted sums in multilayer perceptrons , 1994, IEEE Trans. Neural Networks.

[16]  Stefan C. Kremer,et al.  On the computational power of Elman-style recurrent networks , 1995, IEEE Trans. Neural Networks.

[17]  K. Warwick,et al.  Dynamic recurrent neural network for system identification and control , 1995 .

[18]  Lee A. Feldkamp,et al.  Dynamic neural network methods applied to on-vehicle idle speed control , 1996, Proc. IEEE.

[19]  Steve Rogers,et al.  Adaptive Filter Theory , 1996 .

[20]  Yoshifusa Ito,et al.  Nonlinearity creates linear independence , 1996, Adv. Comput. Math..

[21]  Irwin W. Sandberg,et al.  Uniform approximation of multidimensional myopic maps , 1997 .

[22]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[23]  T. Sejnowski,et al.  Spatial Transformations in the Parietal Cortex Using Basis Functions , 1997, Journal of Cognitive Neuroscience.

[24]  J. A. Kuznecov.  Elements of applied bifurcation theory , 1998 .

[25]  Danil V. Prokhorov,et al.  Enhanced Multi-Stream Kalman Filter Training for Recurrent Networks , 1998 .

[26]  Chi-Sang Poon,et al.  Linear independence of internal representations in multilayer perceptrons , 1999, IEEE Trans. Neural Networks.

[27]  Jose C. Principe.  Dynamic Neural Networks and Optimal Signal Processing , 2001 .

[28]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[29]  Deniz Erdogmus,et al.  Generalized information potential criterion for adaptive system training , 2002, IEEE Trans. Neural Networks.

[30]  Rajvignesh Thogulua Chandrasekar,et al.  INFORMATION THEORETIC SELF-ORGANIZATION OF MULTIPLE AGENTS , 2003 .

[31]  Deniz Erdoğmuş,et al.  Online entropy manipulation: stochastic information gradient , 2003, IEEE Signal Processing Letters.

[32]  Nils Bertschinger,et al.  Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks , 2004, Neural Computation.

[33]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[34]  Justin C. Sanchez.  FROM CORTICAL NEURAL SPIKE TRAINS TO BEHAVIOR: MODELING AND ANALYSIS , 2004 .

[35]  Deniz Erdogmus,et al.  Learning mappings in brain machine interfaces with echo state networks , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[36]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .