Quantitative evaluation of the performance of discrete-time reservoir computers in the forecasting, filtering, and reconstruction of stochastic stationary signals

This paper extends the notion of information processing capacity for non-independent input signals in the context of reservoir computing (RC). The presence of input autocorrelation makes worthwhile the treatment of forecasting and filtering problems for which we explicitly compute this generalized capacity as a function of the reservoir parameter values using a streamlined model. The reservoir model leading to these developments is used to show that, whenever that approximation is valid, this computational paradigm satisfies the so called separation and fading memory properties that are usually associated with good information processing performances. We show that several standard memory, forecasting, and filtering problems that appear in the parametric stochastic time series context can be readily formulated and tackled via RC which, as we show, significantly outperforms standard techniques in some instances.

[1]  Björn Holmquist,et al.  Moments and cumulants of the multivariate normal distribution. , 1988 .

[2]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[3]  Juan-Pablo Ortega,et al.  Hybrid Forecasting with Estimated Temporally Aggregated Linear Processes , 2012 .

[4]  P. Manimaran,et al.  Modelling Financial Time Series , 2006 .

[5]  Laurent Larger,et al.  Nonlinear Memory Capacity of Parallel Time-Delay Reservoir Computers in the Processing of Multidimensional Signals , 2015, Neural Computation.

[6]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[7]  Herbert Jaeger,et al.  Optimization and applications of echo state networks with leaky- integrator neurons , 2007, Neural Networks.

[8]  Kostas Triantafyllopoulos,et al.  On the central moments of the multidimensional Gaussian distribution , 2003 .

[9]  Amir F. Atiya,et al.  New results on recurrent network training: unifying the algorithms and accelerating convergence , 2000, IEEE Trans. Neural Networks Learn. Syst..

[10]  Laurent Larger,et al.  Optimal nonlinear information processing capacity in delay-based reservoir computers , 2014, Scientific Reports.

[11]  Benjamin Schrauwen,et al.  Information Processing Capacity of Dynamical Systems , 2012, Scientific Reports.

[12]  Wolfgang Maass,et al.  Liquid State Machines: Motivation, Theory, and Applications , 2010 .

[13]  J. Wooldridge,et al.  A Capital Asset Pricing Model with Time-Varying Covariances , 1988, Journal of Political Economy.

[14]  J. Ortega,et al.  Asymptotic Forecasting Error Evaluation for Estimated Temporally Aggregated Linear Processes , 2014 .

[15]  J. Nelder,et al.  Hierarchical Generalized Linear Models , 1996 .

[16]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[17]  Benjamin Schrauwen,et al.  Memory in linear recurrent neural networks in continuous time , 2010, Neural Networks.

[18]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[19]  Helmut Ltkepohl,et al.  New Introduction to Multiple Time Series Analysis , 2007 .

[20]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[21]  T. Bollerslev,et al.  Generalized autoregressive conditional heteroskedasticity , 1986 .

[22]  N. Shephard,et al.  Multivariate stochastic variance models , 1994 .

[23]  Richard A. Davis,et al.  Introduction to time series and forecasting , 1998 .

[24]  José Manuel Gutiérrez,et al.  Memory and Nonlinear Mapping in Reservoir Computing with Two Uncoupled Nonlinear Delay Nodes , 2013 .

[25]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[26]  Youngjo Lee,et al.  GLM-methods for volatility models , 2008 .

[27]  Benjamin Schrauwen,et al.  Optoelectronic Reservoir Computing , 2011, Scientific Reports.

[28]  Construction, Management, and Performance of Sparse Markowitz Portfolios , 2012 .

[29]  L Pesquera,et al.  Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. , 2012, Optics express.

[30]  J. Nelder,et al.  Double hierarchical generalized linear models , 2006 .

[31]  Haim Sompolinsky,et al.  Short-term memory in orthogonal neural networks. , 2004, Physical review letters.

[32]  R. Engle Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation , 1982 .

[33]  Nigel Crook Nonlinear transient computation , 2007, Neurocomputing.

[34]  Laurent Larger,et al.  Stochastic Nonlinear Time Series Forecasting Using Time-Delay Reservoir Computers: Performance and Universality , 2013, Neural Networks.

[35]  Daniel Brunner,et al.  Parallel photonic information processing at gigabyte per second data rates using transient states , 2013, Nature Communications.

[36]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[37]  Surya Ganguli,et al.  Memory traces in dynamical systems , 2008, Proceedings of the National Academy of Sciences.

[38]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[39]  M. McAleer,et al.  Stationarity and the existence of moments of a family of GARCH processes , 2002 .

[40]  Michael McAleer,et al.  A Survey of Recent Theoretical Results for Time Series Models with GARCH Errors , 2001 .

[41]  L. Appeltant,et al.  Information processing using a single dynamical node as complex system , 2011, Nature communications.

[42]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[43]  Woojoo Lee,et al.  The hierarchical-likelihood approach to autoregressive stochastic volatility models , 2011, Comput. Stat. Data Anal..

[44]  Stefan J. Kiebel,et al.  Re-visiting the echo state property , 2012, Neural Networks.

[45]  José Manuel Gutiérrez,et al.  Simple reservoirs with chain topology based on a single time-delay nonlinear node , 2012, ESANN.