Embedding and approximation theorems for echo state networks

Echo State Networks (ESNs) are a class of single-layer recurrent neural networks that have enjoyed recent attention. In this paper we prove that a suitable ESN, trained on a series of measurements of an invertible dynamical system, induces a C1 map from the dynamical system's phase space to the ESN's reservoir space. We call this the Echo State Map. We then prove that the Echo State Map is generically an embedding with positive probability. Under additional mild assumptions, we further conjecture that the Echo State Map is almost surely an embedding. For sufficiently large, and specially structured, but still randomly generated ESNs, we prove that there exists a linear readout layer that allows the ESN to predict the next observation of a dynamical system arbitrarily well. Consequently, if the dynamical system under observation is structurally stable then the trained ESN will exhibit dynamics that are topologically conjugate to the future behaviour of the observed dynamical system. Our theoretical results connect the theory of ESNs to the delay-embedding literature for dynamical systems, and are supported by numerical evidence from simulations of the traditional Lorenz equations. The simulations confirm that, from a one dimensional observation function, an ESN can accurately infer a range of geometric and topological features of the dynamics such as the eigenvalues of equilibrium points, Lyapunov exponents and homology groups.

[1]  R. F. Williams,et al.  The structure of Lorenz attractors , 1979 .

[2]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[3]  Robert Jenssen,et al.  Training Echo State Networks with Regularization Through Dimensionality Reduction , 2016, Cognitive Computation.

[4]  Toshiyuki Yamane,et al.  Recent Advances in Physical Reservoir Computing: A Review , 2018, Neural Networks.

[5]  Juan-Pablo Ortega,et al.  Differentiable reservoir computing , 2019, J. Mach. Learn. Res..

[6]  Juan-Pablo Ortega,et al.  Echo state networks are universal , 2018, Neural Networks.

[7]  Lukas Gonon,et al.  Approximation Bounds for Random Neural Networks and Reservoir Systems , 2020, ArXiv.

[8]  Jürgen Hackl.  TikZ-network manual , 2017, ArXiv.

[9]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[10]  Devika Subramanian,et al.  Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network , 2020, Nonlinear Processes in Geophysics.

[11]  Christopher J. Tralie,et al.  Ripser.py: A Lean Persistent Homology Library for Python , 2018, J. Open Source Softw..

[12]  Parlitz,et al.  Generalized synchronization, predictability, and equivalence of unidirectionally coupled dynamical systems. , 1996, Physical review letters.

[13]  F. W. Warner.  Foundations of Differentiable Manifolds and Lie Groups , 1971 .

[14]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[15]  Mikael Vejdemo-Johansson,et al.  javaPlex: A Research Software Package for Persistent (Co)Homology , 2014, ICMS.

[16]  D. Broomhead,et al.  Robust estimation of tangent maps and Liapunov spectra , 1996 .

[17]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[18]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[19]  Zehong Yang,et al.  Short-term stock price prediction based on echo state networks , 2009, Expert Syst. Appl..

[20]  Jaideep Pathak,et al.  Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data. , 2017, Chaos.

[21]  R. Ghrist.  Barcodes: The persistent topology of data , 2007 .

[22]  Jaideep Pathak,et al.  Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics , 2019, Neural Networks.

[23]  J. Sprott.  Chaos and time-series analysis , 2001 .

[24]  Min Han,et al.  Analyzing the state space property of echo state networks for chaotic system prediction , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[25]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[26]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[27]  Thijs Becker,et al.  Bayesian optimization of hyper-parameters in reservoir computing , 2016, ArXiv.

[28]  Mark R. Muldoon,et al.  Topology from time series , 1993 .

[29]  S. Banach.  Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales , 1922 .

[30]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[31]  Stefan J. Kiebel,et al.  Re-visiting the echo state property , 2012, Neural Networks.

[32]  Dianne P. O'Leary,et al.  Deblurring Images: Matrices, Spectra and Filtering , 2006, J. Electronic Imaging.

[33]  H. Hahn.  Sur quelques points du calcul fonctionnel , 1908 .

[34]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[35]  J. Meiss,et al.  Exploring the topology of dynamical reconstructions , 2015, 1506.01128.

[36]  H. Whitney.  The Self-Intersections of a Smooth n-Manifold in 2n-Space , 1944 .

[37]  Paul-Gerhard Plöger,et al.  Echo State Networks for Mobile Robot Modeling and Control , 2003, RoboCup.

[38]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[39]  Stefan Rotter,et al.  Functional identification of biological neural networks using reservoir adaptation for point processes , 2009, Journal of Computational Neuroscience.

[40]  Kyongmin Yeo,et al.  Data-driven Reconstruction of Nonlinear Dynamics from Sparse Observation , 2019, J. Comput. Phys..

[41]  H. Jaeger,et al.  Stepping forward through echoes of the past : forecasting with Echo State Networks , 2007 .

[42]  John G. Harris,et al.  Automatic speech recognition using a predictive echo state network classifier , 2007, Neural Networks.

[43]  Caihong Li,et al.  Multi-steps prediction of chaotic time series based on echo state network , 2010, 2010 IEEE Fifth International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA).

[44]  Benjamin Schrauwen,et al.  An overview of reservoir computing: theory, applications and implementations , 2007, ESANN.

[45]  Michelle Girvan,et al.  Forecasting of Spatio-temporal Chaotic Dynamics with Recurrent Neural Networks: a comparative study of Reservoir Computing and Backpropagation Algorithms , 2019, ArXiv.

[46]  Garrison W. Cottrell,et al.  2007 Special Issue: Learning grammatical structure with Echo State Networks , 2007 .

[47]  Devika Subramanian,et al.  Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM , 2019, ArXiv.

[48]  J. Huke.  Embedding Nonlinear Dynamical Systems: A Guide to Takens' Theorem , 2006 .