A survey on long short-term memory networks for time series prediction

Recurrent neural networks and exceedingly Long short-term memory (LSTM) have been investigated intensively in recent years due to their ability to model and predict nonlinear time-variant system dynamics. The present paper delivers a comprehensive overview of existing LSTM cell derivatives and network architectures for time series prediction. A categorization in LSTM with optimized cell state representations and LSTM with interacting cell states is proposed. The investigated approaches are evaluated against defined requirements being relevant for an accurate time series prediction. These include short-term and long-term memory behavior, the ability for multimodal and multi-step ahead predictions and the according error propagation. Sequence-to-sequence networks with partially conditioning outperform the other approaches, such as bidirectional or associative networks, and are best suited to fulfill the requirements. © 2020 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the 14th CIRP Conference on Intelligent Computation in Manufacturing Engineering.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[3]  Sung-Bae Cho,et al.  Predicting residential energy consumption using CNN-LSTM neural networks , 2019, Energy.

[4]  Seul Jung,et al.  Pedestrian trajectory prediction via the Social‐Grid LSTM model , 2018, The Journal of Engineering.

[5]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[6]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[7]  Ali Akbar Safavi,et al.  Nonlinear and Time Varying System Identification Using a Novel Adaptive Fully Connected Recurrent Wavelet Network , 2019, 2019 27th Iranian Conference on Electrical Engineering (ICEE).

[8]  Thomas S. Huang,et al.  Dilated Recurrent Neural Networks , 2017, NIPS.

[9]  Zhiyong Cui,et al.  Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Prediction , 2018, ArXiv.

[10]  Hang Li,et al.  “ Tony ” DNN Embedding for “ Tony ” Selective Read for “ Tony ” ( a ) Attention-based Encoder-Decoder ( RNNSearch ) ( c ) State Update s 4 SourceVocabulary Softmax Prob , 2016 .

[11]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[12]  Alexandre Boulch,et al.  Deep Sequence-to-Sequence Neural Networks for Ionospheric Activity Map Prediction , 2017, ICONIP.

[13]  Benjamin Lindemann,et al.  Anomaly detection in discrete manufacturing using self-learning approaches , 2019, Procedia CIRP.

[14]  Siu Cheung Hui,et al.  Learning to Rank Question Answer Pairs with Holographic Dual LSTM Architecture , 2017, SIGIR.

[15]  Quoc V. Le,et al.  Massive Exploration of Neural Machine Translation Architectures , 2017, EMNLP.

[16]  Shi-Jinn Horng,et al.  Time Series Forecasting Using Sequence-to-Sequence Deep Learning Framework , 2018, 2018 9th International Symposium on Parallel Architectures, Algorithms and Programming (PAAP).

[17]  Vadim V. Strijov,et al.  Position-Based Content Attention for Time Series Forecasting with Sequence-to-Sequence RNNs , 2017, ICONIP.

[18]  Chuan Ding,et al.  Parallel Architecture of Convolutional Bi-Directional LSTM Neural Networks for Network-Wide Metro Ridership Prediction , 2019, IEEE Transactions on Intelligent Transportation Systems.

[19]  Ping-Huan Kuo,et al.  A Deep CNN-LSTM Model for Particulate Matter (PM2.5) Forecasting in Smart Cities , 2018, Sensors.

[20]  Daniel L. Marino,et al.  Building energy load forecasting using Deep Neural Networks , 2016, IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society.

[21]  Samy Bengio,et al.  An Online Sequence-to-Sequence Model Using Partial Conditioning , 2015, NIPS.

[22]  Bernhard Sick,et al.  Deep Learning for solar power forecasting — An approach using AutoEncoder and LSTM Neural Networks , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[23]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[24]  Alex Graves,et al.  Grid Long Short-Term Memory , 2015, ICLR.

[25]  Garrison W. Cottrell,et al.  A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction , 2017, IJCAI.

[26]  Michael I. Jordan.  Attractor dynamics and parallelism in a connectionist sequential machine , 1990 .

[27]  Benjamin Lindemann,et al.  Detektion von Anomalien zur Qualitätssicherung basierend auf Sequence-to-Sequence LSTM Netzen , 2019, Autom..

[28]  Alex Graves,et al.  Associative Long Short-Term Memory , 2016, ICML.

[29]  Mark Reynolds,et al.  SS-LSTM: A Hierarchical LSTM Model for Pedestrian Trajectory Prediction , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[30]  Daniel Hsu,et al.  Multi-period Time Series Modeling with Sparsity via Bayesian Variational Inference , 2017 .

[31]  Pietro Liò,et al.  Cross-modal Recurrent Models for Weight Objective Prediction from Multimodal Time-series Data , 2017, PervasiveHealth.

[32]  Victor O. K. Li,et al.  Deep Multi-Scale Convolutional LSTM Network for Travel Demand and Origin-Destination Predictions , 2020, IEEE Transactions on Intelligent Transportation Systems.

[33]  Ruben Villegas,et al.  Learning to Generate Long-term Future via Hierarchical Prediction , 2017, ICML.

[34]  Jialin Jiang,et al.  Hierarchical Gated Recurrent Unit with Semantic Attention for Event Prediction , 2020, Future Internet.