Performance enhancing techniques for deep learning models in time series forecasting

Abstract Time series forecasting uses deterministic algorithms to capture past temporal information or dependencies that can be used to predict future patterns. Studies have shown that traditional forecasting techniques are outperformed by deep learning models. Since then research work has been much focused on proposing different network models, little attention has been paid to improve the performance of existing models. In this paper, we compare the performance of several existing deep learning models used in both single and multiple time series forecasting tasks. We then propose two different approaches to improve the models’ performance. Specifically, we present a fine-grained attention mechanism that achieves a much better performance for multi-step forecasting tasks. An ensemble technique is then proposed to further improve the performance of all the models.

[1]  Meng Joo Er,et al.  NARMAX time series model prediction: feedforward and recurrent fuzzy neural network approaches , 2005, Fuzzy Sets Syst..

[2]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[3]  Guy Pujolle,et al.  A Long Short-Term Memory Recurrent Neural Network Framework for Network Traffic Matrix Prediction , 2017, ArXiv.

[4]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[5]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[6]  Amir F. Atiya,et al.  A Bias and Variance Analysis for Multistep-Ahead Time Series Forecasting , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Tao Lin,et al.  Hybrid Neural Networks for Learning the Trend in Time Series , 2017, IJCAI.

[8]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[9]  Bin Yang,et al.  Correlated Time Series Forecasting using Multi-Task Deep Neural Networks , 2018, CIKM.

[10]  Ji-Rong Wen,et al.  Temporal Embedding in Convolutional Neural Networks for Robust Learning of Abstract Snippets , 2015, ArXiv.

[11]  Germain Forestier,et al.  Deep learning for time series classification: a review , 2018, Data Mining and Knowledge Discovery.

[12]  Zhongyi Hu,et al.  Multi-step-ahead time series prediction using multiple-output support vector regression , 2014, Neurocomputing.

[13]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[14]  Durdu Ömer Faruk.  A hybrid neural network and ARIMA model for water quality time series prediction , 2010, Eng. Appl. Artif. Intell..

[15]  Zhiyong Cui,et al.  Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Prediction , 2018, ArXiv.

[16]  Vivek Srikumar,et al.  Predicting electricity consumption for commercial and residential buildings using deep recurrent neural networks , 2018 .

[17]  Ali Ouni,et al.  Optimal Deep Learning LSTM Model for Electric Load Forecasting using Feature Selection and Genetic Algorithm: Comparison with Machine Learning Approaches † , 2018, Energies.

[18]  Wang Xiangxue,et al.  Data-Driven Short-Term Forecasting for Urban Road Network Traffic Based on Data Processing and LSTM-RNN , 2018, Arabian Journal for Science and Engineering.

[19]  Shouhuai Xu,et al.  A deep learning framework for predicting cyber attacks rates , 2019, EURASIP J. Inf. Secur..

[20]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[21]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[22]  Yoshua Bengio,et al.  Fine-grained attention mechanism for neural machine translation , 2018, Neurocomputing.

[23]  Garrison W. Cottrell,et al.  Substructure Vibration NARX Neural Network Approach for Statistical Damage Inference , 2013 .

[24]  John Cristian Borges Gamboa,et al.  Deep Learning for Time-Series Analysis , 2017, ArXiv.

[25]  Mohit Gupta,et al.  Improving Time Series Forecasting Using Mathematical and Deep Learning Models , 2018, BDA.

[26]  Garrison W. Cottrell,et al.  A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction , 2017, IJCAI.

[27]  Jeffrey L. Elman,et al.  Distributed Representations, Simple Recurrent Networks, and Grammatical Structure , 1991, Mach. Learn..

[28]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[29]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[30]  Daniel L. Marino,et al.  Building energy load forecasting using Deep Neural Networks , 2016, IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society.

[31]  Ronald Davis,et al.  Neural networks and deep learning , 2017 .

[32]  P. Whittle,et al.  TESTS OF FIT IN TIME SERIES , 1952 .

[33]  Juan Pardo,et al.  On-line learning of indoor temperature forecasting models towards energy efficiency , 2014 .

[34]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[35]  Spyros Makridakis,et al.  ARMA Models and the Box–Jenkins Methodology , 1997 .

[36]  Jian Zhang,et al.  Robust spike-and-slab deep Boltzmann machines for face denoising , 2018, Neural Computing and Applications.