Nonstationary Time Series Prediction Based on Deep Echo State Network Tuned by Bayesian Optimization

The predictions from time series data can help us sense development trends and make scientific decisions in advance. The commonly used forecasting methods with backpropagation consume a lot of computational resources. The deep echo state network (DeepESN) is an advanced prediction method with a deep neural network structure and training algorithm without backpropagation. In this paper, a Bayesian optimization algorithm (BOA) is proposed to optimize DeepESN to address the problem of increasing parameter scale. Firstly, the DeepESN was studied and constructed as the basic prediction model for the time series data. Secondly, the BOA was reconstructed, based on the DeepESN, for optimal parameter searching. The algorithm is proposed within the framework of the DeepESN. Thirdly, an experiment was conducted to verify the DeepESN with a BOA within three datasets: simulation data generated from computer programs, a real humidity dataset collected from Beijing, and a power load dataset obtained from America. Compared with the models of BP (backpropagation), LSTM (long short-term memory), GRU (gated recurrent unit), and ESN (echo state network), DeepESN obtained optimal results, which were 0.0719, 18.6707, and 764.5281 using RMSE evaluation. While getting better accuracy, the BOA optimization time was only 323.4 s, 563.2 s, and 9854 s for the three datasets. It is more efficient than grid search and grey wolf optimizer.

[1]  Yanghua Xiao,et al.  A Fine-Grained Recognition Neural Network with High-Order Feature Maps via Graph-Based Embedding for Natural Bird Diversity Conservation , 2023, International journal of environmental research and public health.

[2]  Tingli Su,et al.  BMAE-Net: A Data-Driven Weather Prediction Network for Smart Agriculture , 2023, Agronomy.

[3]  P. Chakrabarti,et al.  Variational Bayesian Network with Information Interpretability Filtering for Air Quality Forecasting , 2023, Mathematics.

[4]  P. Chakrabarti,et al.  Deep Spatio-Temporal Graph Network with Self-Optimization for Air Quality Prediction , 2023, Entropy.

[5]  Leandro dos Santos Coelho,et al.  Novel hybrid model based on echo state neural network applied to the prediction of stock price return volatility , 2021, Expert Syst. Appl..

[6]  Rocío Pérez de Prado,et al.  Long Short-Term Memory Network-Based Metaheuristic for Effective Electric Energy Consumption Prediction , 2021, Applied Sciences.

[7]  Tim Leung,et al.  Financial time series analysis and forecasting with Hilbert–Huang transform feature generation and machine learning , 2021 .

[8]  Jianlong Xu,et al.  FM-GRU: A Time Series Prediction Method for Water Quality Based on seq2seq Framework , 2021, Water.

[9]  M. C. Lineesh.  Time series analysis and forecasting of air quality index , 2021 .

[10]  Marcin Woźniak,et al.  Red fox optimization algorithm , 2021, Expert Syst. Appl..

[11]  Marcin Woźniak,et al.  Short-Term Load Forecasting Based on Adabelief Optimized Temporal Convolutional Network and Gated Recurrent Unit Hybrid Neural Network , 2021, IEEE Access.

[12]  Song-hee Kim,et al.  Comparison Analysis of Treatment Methods and ARIMA Time-Series Forecasting of Basic Water Components in Effluent from Small-scale Public Sewage Treatment Facilities , 2020, Journal of the Korean Society for Environmental Technology.

[13]  Dong Liu,et al.  GRU-corr Neural Network Optimized by Improved PSO Algorithm for Time Series Prediction , 2020, Int. J. Artif. Intell. Tools.

[14]  Mehmet Özger,et al.  Comparison of wavelet and empirical mode decomposition hybrid models in drought prediction , 2020, Comput. Electron. Agric..

[15]  Marcin Woźniak,et al.  Prediction of Streamflow Based on Dynamic Sliding Window LSTM , 2020, Water.

[16]  Xiaoyi Wang,et al.  A health performance evaluation method of multirotors under wind turbulence , 2020, Nonlinear Dynamics.

[17]  Ranjan Kumar Behera,et al.  Comparative Study of Real Time Machine Learning Models for Stock Prediction through Streaming Data , 2020, J. Univers. Comput. Sci..

[18]  Amir H. Gandomi,et al.  Marine Predators Algorithm: A nature-inspired metaheuristic , 2020, Expert Syst. Appl..

[19]  Leandro dos Santos Coelho,et al.  Forecasting Brazilian and American COVID-19 cases based on artificial intelligence coupled with climatic exogenous variables , 2020, Chaos, Solitons & Fractals.

[20]  Brian R. King,et al.  Time series prediction using deep echo state networks , 2020, Neural Computing and Applications.

[21]  Y. Akdi,et al.  Estimation and forecasting of PM10 air pollution in Ankara via time series and harmonic regressions , 2020, International Journal of Environmental Science and Technology.

[22]  Sungwon Kim,et al.  Deep echo state network: a novel machine learning approach to model dew point temperature using meteorological variables , 2020 .

[23]  Su Yang,et al.  An echo state network architecture based on quantum logic gate and its optimization , 2020, Neurocomputing.

[24]  Sukono,et al.  Indonesian Rupiah Exchange Rate in Facing COVID-19 (A Time Series-Machine Learning Approach) , 2020 .

[25]  Dongxiao Niu,et al.  Wind Power Short-Term Forecasting Hybrid Model Based on CEEMD-SE Method , 2019, Processes.

[26]  Terrence L. Chambers,et al.  Hour-Ahead Solar Irradiance Forecasting Using Multivariate Gated Recurrent Units , 2019, Energies.

[27]  Yining Wang,et al.  The Forecasting of PM2.5 Using a Hybrid Model Based on Wavelet Transform and an Improved Deep Learning Algorithm , 2019, IEEE Access.

[28]  Chii-Chang Chen,et al.  Application of the deep learning for the prediction of rainfall in Southern Taiwan , 2019, Scientific Reports.

[29]  YONGBO LIAO,et al.  Deep echo state network with reservoirs of multiple activation functions for time-series prediction , 2019, Sādhanā.

[30]  Christopher K. Wikle,et al.  Deep echo state networks with uncertainty quantification for spatio‐temporal forecasting , 2018, Environmetrics.

[31]  Yan Chen,et al.  A Novel Dual-Scale Deep Belief Network Method for Daily Urban Water Demand Forecasting , 2018 .

[32]  Norbert A. Agana,et al.  EMD-Based Predictive Deep Belief Network for Time Series Prediction: An Application to Drought Forecasting , 2018 .

[33]  Qiang Liu,et al.  Financial time series prediction using ℓ2, 1RF-ELM , 2018, Neurocomputing.

[34]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[35]  Claudio Gallicchio,et al.  Deep reservoir computing: A critical experimental analysis , 2017, Neurocomputing.

[36]  Yulei Rao,et al.  A deep learning framework for financial time series using stacked autoencoders and long-short term memory , 2017, PloS one.

[37]  Amir Hussain,et al.  Multilayered Echo State Machine: A Novel Architecture and Algorithm , 2017, IEEE Transactions on Cybernetics.

[38]  Jaeseok Choi,et al.  Hourly Average Wind Speed Simulation and Forecast Based on ARMA Model in Jeju Island, Korea , 2016 .

[39]  M. Hadi Amini,et al.  ARIMA-based decoupled time series forecasting of electric vehicle charging demand for stochastic power system operation , 2016 .

[40]  Herman Eerens,et al.  Image time series processing for agriculture monitoring , 2014, Environ. Model. Softw..

[41]  Andrew Lewis,et al.  Grey Wolf Optimizer , 2014, Adv. Eng. Softw..

[42]  Jun Wang,et al.  Chaotic Time Series Prediction Based on a Novel Robust Echo State Network , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[43]  Jennifer N. Hird,et al.  Noise reduction of NDVI time series: An empirical comparison of selected techniques , 2009 .

[44]  J. Torres,et al.  Forecast of hourly average wind speed with ARMA models in Navarre (Spain) , 2005 .

[45]  Guoqiang Peter Zhang,et al.  Time series forecasting using a hybrid ARIMA and neural network model , 2003, Neurocomputing.

[46]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[47]  J. Contreras,et al.  ARIMA Models to Predict Next-Day Electricity Prices , 2002, IEEE Power Engineering Review.

[48]  Jürgen Schmidhuber,et al.  LSTM can Solve Hard Long Time Lag Problems , 1996, NIPS.

[49]  James Cadzow,et al.  ARMA Time Series Modeling: an Effective Method , 1983, IEEE Transactions on Aerospace and Electronic Systems.