On the Statistical Challenges of Echo State Networks and Some Potential Remedies

Echo state networks are powerful recurrent neural networks. However, they are often unstable and shaky, making the process of finding an good ESN for a specific dataset quite hard. Obtaining a superb accuracy by using the Echo State Network is a challenging task. We create, develop and implement a family of predictably optimal robust and stable ensemble of Echo State Networks via regularizing the training and perturbing the input. Furthermore, several distributions of weights have been tried based on the shape to see if the shape of the distribution has the impact for reducing the error. We found ESN can track in short term for most dataset, but it collapses in the long run. Short-term tracking with large size reservoir enables ESN to perform strikingly with superior prediction. Based on this scenario, we go a further step to aggregate many of ESNs into an ensemble to lower the variance and stabilize the system by stochastic replications and bootstrapping of input data.

[1]  Jiabin Wang,et al.  Functional echo state network for time series classification , 2016, Inf. Sci..

[2]  Eric R. Ziegel,et al.  Data: A Collection of Problems From Many Fields for the Student and Research Worker , 1987 .

[3]  Boris Bacic,et al.  Echo State Network Ensemble for Human Motion Data Temporal Phasing: A Case Study on Tennis Forehands , 2016, ICONIP.

[4]  Rik Van de Walle,et al.  Real-Time Reservoir Computing Network-Based Systems for Detection Tasks on Visual Contents , 2015, 2015 7th International Conference on Computational Intelligence, Communication Systems and Networks.

[5]  Ashley A. Prater,et al.  Comparison of echo state network output layer classification methods on noisy data , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[6]  Mantas Lukosevicius,et al.  A Practical Guide to Applying Echo State Networks , 2012, Neural Networks: Tricks of the Trade.

[7]  Christopher K. Wikle,et al.  An ensemble quadratic echo state network for non‐linear spatio‐temporal forecasting , 2017, 1708.05094.

[8]  A. M. Walker,et al.  A Survey of statistical work on the Mackenzie River series of annual Canadian lynx trappings for the , 1977 .

[9]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[10]  D. F. Andrews,et al.  Data : a collection of problems from many fields for the student and research worker , 1985 .

[11]  J. W. Taylor,et al.  Short-term electricity demand forecasting using double seasonal exponential smoothing , 2003, J. Oper. Res. Soc..

[12]  Cory Merkel,et al.  Design and Analysis of a Neuromemristive Reservoir Computing Architecture for Biosignal Processing , 2016, Front. Neurosci..

[13]  Benjamin Schrauwen,et al.  Connectivity, Dynamics, and Memory in Reservoir Computing with Binary and Analog Neurons , 2010, Neural Computation.

[14]  Benjamin Schrauwen,et al.  Real-Time Epileptic Seizure Detection on Intra-cranial Rat Data Using Reservoir Computing , 2008, ICONIP.

[15]  Ronald L. Rivest,et al.  On Estimating the Size and Confidence of a Statistical Audit , 2007, EVT.

[16]  Raphaël Couturier,et al.  Echo State Networks-Based Reservoir Computing for MNIST Handwritten Digits Recognition , 2016, 2016 IEEE Intl Conference on Computational Science and Engineering (CSE) and IEEE Intl Conference on Embedded and Ubiquitous Computing (EUC) and 15th Intl Symposium on Distributed Computing and Applications for Business Engineering (DCABES).

[17]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[18]  Richard A. Davis,et al.  Model Building and Forecasting with ARIMA Processes , 1991 .

[19]  Geoffrey E. Hinton,et al.  Temporal-Kernel Recurrent Neural Networks , 2010, Neural Networks.

[20]  Bozhkov Lachezar,et al.  Echo State Network , 2017, Encyclopedia of Machine Learning and Data Mining.