Design of sparse Bayesian echo state network for time series prediction

Echo state network (ESN) refers to a novel recurrent neural network with a largely and randomly generated reservoir and a trainable output layer, which has been utilized in the time series prediction. In spite of that, since the output weights are computed by the simple linear regression, there may be an ill-posed problem in the training process for ESN. In order to tackle this issue, a sparse Bayesian ESN (SBESN) is given. The proposed SBESN attempts to estimate the probability of the outputs and trains the network through sparse Bayesian learning, where independent regularization priors should be implied to each weight rather than sharing one prior for all weights. Simulation results illustrate that the SBESN model is insensitivity to reservoir size and completely outperforms other models.

[1]  Junfei Qiao,et al.  Adaptive lasso echo state network based on modified Bayesian information criterion for nonlinear system modeling , 2019, Neural Computing and Applications.

[2]  Qiong Wu,et al.  Recursive Bayesian echo state network with an adaptive inflation factor for temperature prediction , 2016, Neural Computing and Applications.

[3]  Rohitash Chandra,et al.  Competition and Collaboration in Cooperative Coevolution of Elman Recurrent Neural Networks for Time-Series Prediction , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Hendrik Van Brussel,et al.  Pruning and regularization in reservoir computing , 2009, Neurocomputing.

[5]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[6]  Henry Leung,et al.  Data-driven based model for flow prediction of steam system in steel industry , 2012, Inf. Sci..

[7]  Babita Majhi,et al.  Financial time series prediction using distributed machine learning techniques , 2017, Neural Computing and Applications.

[8]  H. Vincent Poor,et al.  Regularized Variational Bayesian Learning of Echo State Networks with Delay&Sum Readout , 2012, Neural Computation.

[9]  Chi-Man Vong,et al.  Sparse Bayesian Extreme Learning Machine for Multi-classification , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Antonio J. Serrano,et al.  BELM: Bayesian Extreme Learning Machine , 2011, IEEE Transactions on Neural Networks.

[11]  Arash Miranian,et al.  Developing a Local Least-Squares Support Vector Machines-Based Neuro-Fuzzy Model for Nonlinear and Chaotic Time Series Prediction , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Michael E. Tipping.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[13]  Bhim Singh,et al.  Implementation of Echo-State Network-Based Control for Power Quality Improvement , 2017, IEEE Transactions on Industrial Electronics.

[14]  Xuefeng Yan,et al.  Improved simple deterministically constructed Cycle Reservoir Network with Sensitive Iterative Pruning Algorithm , 2014, Neurocomputing.

[15]  Claudio Gallicchio,et al.  Architectural and Markovian factors of echo state networks , 2011, Neural Networks.

[16]  Wing W. Y. Ng,et al.  Convolutional Multitimescale Echo State Network , 2019, IEEE Transactions on Cybernetics.

[17]  Min Han,et al.  Adaptive Elastic Echo State Network for Multivariate Time Series Prediction , 2016, IEEE Transactions on Cybernetics.

[18]  Michael E. Tipping.  Bayesian Inference: An Introduction to Principles and Practice in Machine Learning , 2003, Advanced Lectures on Machine Learning.

[19]  Junfei Qiao,et al.  Dynamical regularized echo state network for time series prediction , 2019, Neural Computing and Applications.

[20]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[21]  Junfei Qiao,et al.  Growing Echo-State Network With Multiple Subreservoirs , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Jochen J. Steil,et al.  Regularization and stability in reservoir networks with output feedback , 2012, Neurocomputing.

[23]  Stefan J. Kiebel,et al.  Re-visiting the echo state property , 2012, Neural Networks.

[24]  Jinhua Li,et al.  A novel double incremental learning algorithm for time series prediction , 2019, Neural Computing and Applications.

[25]  Chi-Man Vong,et al.  Sparse Bayesian extreme learning machine and its application to biofuel engine performance prediction , 2015, Neurocomputing.

[26]  Dawei Chen,et al.  Research on Traffic Flow Prediction in the Big Data Environment Based on the Improved RBF Neural Network , 2017, IEEE Transactions on Industrial Informatics.

[27]  Qiang Chen,et al.  Adaptive echo state network control for a class of pure-feedback systems with input and output constraints , 2018, Neurocomputing.

[28]  Jun Wang,et al.  Chaotic Time Series Prediction Based on a Novel Robust Echo State Network , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[29]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[30]  Wei Zhang,et al.  An improved kernel-based incremental extreme learning machine with fixed budget for nonstationary time series prediction , 2017, Neural Computing and Applications.

[31]  Yiannis Demiris,et al.  Echo State Gaussian Process , 2011, IEEE Transactions on Neural Networks.

[32]  Lei Wang,et al.  Design of polynomial echo state networks for time series prediction , 2018, Neurocomputing.

[33]  Shu-Xian Lun,et al.  A new echo state network with variable memory length , 2016, Inf. Sci..

[34]  Xiaohua Wang,et al.  Echo State Networks With Orthogonal Pigeon-Inspired Optimization for Image Restoration , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Huaguang Zhang,et al.  Prediction and identification of discrete-time dynamic nonlinear systems based on adaptive echo state network , 2019, Neural Networks.

[36]  Christopher M. Bishop,et al.  Current address: Microsoft Research, , 2022 .

[37]  Zhigang Zeng,et al.  A novel echo state network for multivariate and nonlinear time series prediction , 2018, Appl. Soft Comput..

[38]  Min Han,et al.  Laplacian Echo State Network for Multivariate Time Series Prediction , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Junfei Qiao,et al.  Modeling of nonlinear systems using the self-organizing fuzzy neural network with adaptive gradient algorithm , 2017, Neurocomputing.

[40]  Enrico Zio,et al.  Two Machine Learning Approaches for Short-Term Wind Speed Time-Series Prediction , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[41]  Zheng Liu,et al.  Data-Driven Multiobjective Predictive Control for Wastewater Treatment Process , 2020, IEEE Transactions on Industrial Informatics.

[42]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[43]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[44]  Peter Michael Young,et al.  A tighter bound for the echo state property , 2006, IEEE Transactions on Neural Networks.

[45]  Junfei Qiao,et al.  Online sequential echo state network with sparse RLS algorithm for time series prediction , 2019, Neural Networks.