Temporal pattern attention for multivariate time series forecasting
Abstract:Forecasting of multivariate time series data, for instance the prediction of electricity consumption, solar power production, and polyphonic piano pieces, has numerous valuable applications. However, complex and non-linear interdependencies between time steps and series complicate this task. To obtain accurate prediction, it is crucial to model long-term dependency in time series data, which can be achieved by recurrent neural networks (RNNs) with an attention mechanism. The typical attention mechanism reviews the information at each previous time step and selects relevant information to help generate the outputs; however, it fails to capture temporal patterns across multiple time steps. In this paper, we propose using a set of filters to extract time-invariant temporal patterns, similar to transforming time series data into its “frequency domain”. Then we propose a novel attention mechanism to select relevant time series, and use its frequency domain information for multivariate forecasting. We apply the proposed model on several real-world tasks and achieve state-of-the-art performance in almost all of cases. Our source code is available at https://github.com/gantheory/TPA-LSTM.
暂无分享，去 创建一个
[1] Kyoung-jae Kim,et al. Financial time series forecasting using support vector machines , 2003, Neurocomputing.
[2] Christopher D. Manning,et al. Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.
[3] C. Chatfield,et al. Fourier Analysis of Time Series: An Introduction , 1977, IEEE Transactions on Systems, Man, and Cybernetics.
[4] Ashu Jain,et al. Hybrid neural network models for hydrologic time series forecasting , 2007, Appl. Soft Comput..
[5] Ching-Hua Chuan,et al. Modeling Temporal Tonal Relations in Polyphonic Music Through Deep Networks With a Novel Image-Based Representation , 2018, AAAI.
[6] Yoshua Bengio,et al. Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription , 2012, ICML.
[7] Roger Frigola,et al. Bayesian Time Series Learning with Gaussian Processes , 2015 .
[8] Yoshua Bengio,et al. Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.
[9] Guoqiang Peter Zhang,et al. Time series forecasting using a hybrid ARIMA and neural network model , 2003, Neurocomputing.
[10] Jürgen Schmidhuber,et al. Long Short-Term Memory , 1997, Neural Computation.
[11] Takayuki Osogami,et al. Nonlinear Dynamic Boltzmann Machines for Time-Series Prediction , 2017, AAAI.
[12] Yoshua Bengio,et al. On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.
[13] N. Huang,et al. The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis , 1998, Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.
[14] Carl E. Rasmussen,et al. Integrated pre-processing for Bayesian nonlinear system identification with Gaussian processes , 2013, 52nd IEEE Conference on Decision and Control.
[15] Jasper Snoek,et al. Spectral Representations for Convolutional Neural Networks , 2015, NIPS.
[16] Yoshua Bengio,et al. Convolutional networks for images, speech, and time series , 1998 .
[17] Garrison W. Cottrell,et al. A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction , 2017, IJCAI.
[18] H. Tong,et al. Threshold Autoregression, Limit Cycles and Cyclical Data , 1980 .
[19] Yi-Hsuan Yang,et al. MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment , 2017, AAAI.
[20] ImageNet Classification with Deep Convolutional Neural , 2013 .
[21] Quoc V. Le,et al. Sequence to Sequence Learning with Neural Networks , 2014, NIPS.
[22] Jeffrey L. Elman,et al. Finding Structure in Time , 1990, Cogn. Sci..
[23] S Roberts,et al. Gaussian processes for time-series modelling , 2013, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.
[24] Geoffrey E. Hinton,et al. Learning representations by back-propagating errors , 1986, Nature.
[25] Michael Y. Hu,et al. Forecasting with artificial neural networks: The state of the art , 1997 .
[26] Guokun Lai,et al. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks , 2017, SIGIR.
[27] Geoffrey E. Hinton,et al. Learning representations of back-propagation errors , 1986 .
[28] Geoffrey E. Hinton,et al. ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.
[29] Yoshua Bengio,et al. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.
[30] Colin Raffel,et al. Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching , 2016 .
[31] Alexander J. Smola,et al. Support Vector Method for Function Approximation, Regression Estimation and Signal Processing , 1996, NIPS.
[32] Yoshua Bengio,et al. High-dimensional sequence transduction , 2012, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.
[33] Francis Eng Hock Tay,et al. Support vector machine with adaptive parameters in financial time series forecasting , 2003, IEEE Trans. Neural Networks.
[34] P. Young,et al. Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.
[35] Paul J. Werbos,et al. Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.
[36] Les E. Atlas,et al. Recurrent Networks and NARMA Modeling , 1991, NIPS.
[37] Sheng Chen,et al. NARX-Based Nonlinear System Identification Using Orthogonal Least Squares Basis Hunting , 2008, IEEE Transactions on Control Systems Technology.
[38] Gwilym M. Jenkins,et al. Time series analysis, forecasting and control , 1971 .
[39] Abdelhamid Bouchachia,et al. Ensemble Learning for Time Series Prediction , 2008 .