DSTP-RNN: a dual-stage two-phase attention-based recurrent neural networks for long-term and multivariate time series prediction

Long-term prediction of multivariate time series is still an important but challenging problem. The key to solve this problem is to capture the spatial correlations at the same time, the spatio-temporal relationships at different times and the long-term dependence of the temporal relationships between different series. Attention-based recurrent neural networks (RNN) can effectively represent the dynamic spatio-temporal relationships between exogenous series and target series, but it only performs well in one-step time prediction and short-term time prediction. In this paper, inspired by human attention mechanism including the dual-stage two-phase (DSTP) model and the influence mechanism of target information and non-target information, we propose DSTP-based RNN (DSTP-RNN) and DSTP-RNN-2 respectively for long-term time series prediction. Specifically, we first propose the DSTP-based structure to enhance the spatial correlations between exogenous series. The first phase produces violent but decentralized response weight, while the second phase leads to stationary and concentrated response weight. Secondly, we employ multiple attentions on target series to boost the long-term dependence. Finally, we study the performance of deep spatial attention mechanism and provide experiment and interpretation. Our methods outperform nine baseline methods on four datasets in the fields of energy, finance, environment and medicine, respectively.

[1]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[2]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[3]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[4]  Yuanzhi Li,et al.  A Convergence Theory for Deep Learning via Over-Parameterization , 2018, ICML.

[5]  Luis M. Candanedo,et al.  Data driven prediction models of energy use of appliances in a low-energy house , 2017 .

[6]  Alexander J. Smola,et al.  Stacked Attention Networks for Image Question Answering , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Min Han,et al.  Laplacian Echo State Network for Multivariate Time Series Prediction , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Kunihiko Fukushima,et al.  Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition , 1982 .

[9]  Phil Blunsom,et al.  Recurrent Continuous Translation Models , 2013, EMNLP.

[10]  Junjie Yao,et al.  Forecasting Wavelet Transformed Time Series with Attentive Neural Networks , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[11]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[12]  SMFernandez Fraga,et al.  SCREEN TASK EXPERIMENTS FOR EEG SIGNALS BASED ON SSVEP BRAIN COMPUTER INTERFACE. , 2018 .

[13]  Erol Egrioglu,et al.  Single Multiplicative Neuron Model Artificial Neural Network with Autoregressive Coefficient for Time Series Modelling , 2017, Neural Processing Letters.

[14]  Erol Egrioglu,et al.  Probabilistic forecasting, linearity and nonlinearity hypothesis tests with bootstrapped linear and nonlinear artificial neural network , 2019, J. Exp. Theor. Artif. Intell..

[15]  Yanmin Zhu,et al.  Stock Price Prediction Using Attention-based Multi-Input LSTM , 2018, ACML.

[16]  Jinsong Su,et al.  Neural Machine Translation with Deep Attention , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Tao Lin,et al.  An interpretable LSTM neural network for autoregressive exogenous model , 2018, ICLR.

[18]  Jimeng Sun,et al.  RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism , 2016, NIPS.

[19]  Jing Huang,et al.  Interpretable Convolutional Neural Networks with Dual Local and Global Attention for Review Rating Prediction , 2017, RecSys.

[20]  Jian Liu,et al.  Hierarchical Attention-Based Recurrent Highway Networks for Time Series Prediction , 2018, ArXiv.

[21]  Qian Zhang,et al.  Attention-based recurrent neural networks for accurate short-term and long-term dissolved oxygen prediction , 2019, Comput. Electron. Agric..

[22]  Jin-Hua She,et al.  A hybrid time series prediction model based on recurrent neural network and double joint linear-nonlinear extreme learning network for prediction of carbon efficiency in iron ore sintering process , 2017, Neurocomputing.

[23]  Tao Lin,et al.  Multi-variable LSTM neural network for autoregressive exogenous model , 2018, ArXiv.

[24]  A. Geetha,et al.  Time-series modelling and forecasting: modelling of rainfall prediction using ARIMA model , 2016 .

[25]  M. Steinhauser,et al.  A dual-stage two-phase model of selective attention. , 2010, Psychological review.

[26]  Ming Li,et al.  Forecasting Fine-Grained Air Quality Based on Big Data , 2015, KDD.

[27]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[28]  Deyi Xiong,et al.  A GRU-Gated Attention Model for Neural Machine Translation , 2017, ArXiv.

[29]  Antoine J.-P. Tixier,et al.  Notes on Deep Learning for NLP , 2018, ArXiv.

[30]  Johan A. K. Suykens,et al.  Financial time series prediction using least squares support vector machines within the evidence framework , 2001, IEEE Trans. Neural Networks.

[31]  Yao Zhao,et al.  EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction , 2018, Knowl. Based Syst..

[32]  Monidipa Das,et al.  FB-STEP: A fuzzy Bayesian network based data-driven framework for spatio-temporal prediction of climatological time series data , 2019, Expert Syst. Appl..

[33]  Juan Pardo,et al.  On-line learning of indoor temperature forecasting models towards energy efficiency , 2014 .

[34]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[35]  J. Michael Herrmann,et al.  Lagged correlation-based deep learning for directional trend change prediction in financial time series , 2018, Expert Syst. Appl..

[36]  Richard Hull,et al.  Correcting Forecasts with Multifactor Neural Attention , 2016, ICML.

[37]  Garrison W. Cottrell,et al.  A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction , 2017, IJCAI.

[38]  Shou-De Lin,et al.  A Memory-Network Based Solution for Multivariate Time-Series Forecasting , 2018, ArXiv.

[39]  M. Hadi Amini,et al.  ARIMA-based decoupled time series forecasting of electric vehicle charging demand for stochastic power system operation , 2016 .

[40]  Alex Graves,et al.  Long Short-Term Memory , 2020, Computer Vision.

[41]  Raman Arora,et al.  Understanding Deep Neural Networks with Rectified Linear Units , 2016, Electron. Colloquium Comput. Complex..

[42]  Éric Gaussier,et al.  Period-aware content attention RNNs for time series forecasting with missing values , 2018, Neurocomputing.

[43]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[44]  Enrico Zio,et al.  SVM hyperparameters tuning for recursive multi-step-ahead prediction , 2017, Neural Computing and Applications.

[45]  Zhiqiang Ge,et al.  Scalable Semisupervised GMM for Big Data Quality Prediction in Multimode Processes , 2019, IEEE Transactions on Industrial Electronics.

[46]  Marcello Federico,et al.  Deep Neural Machine Translation with Weakly-Recurrent Units , 2018, EAMT.

[47]  Shiguang Shan,et al.  Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism , 2019, IEEE Transactions on Image Processing.

[48]  Shyamala Sivakumar,et al.  Marginally Stable Triangular Recurrent Neural Network Architecture for Time Series Prediction , 2018, IEEE Transactions on Cybernetics.

[49]  Yu Zheng,et al.  GeoMAN: Multi-level Attention Networks for Geo-sensory Time Series Prediction , 2018, IJCAI.

[50]  Lian Zhou,et al.  A Hierarchical Multimodal Attention-based Neural Network for Image Captioning , 2017, SIGIR.

[51]  Vadim V. Strijov,et al.  Position-Based Content Attention for Time Series Forecasting with Sequence-to-Sequence RNNs , 2017, ICONIP.

[52]  Yu Zheng,et al.  Urban Water Quality Prediction Based on Multi-Task Multi-View Learning , 2016, IJCAI.

[53]  Zhihua Zhang,et al.  Attention-via-Attention Neural Machine Translation , 2018, AAAI.