DTDR-ALSTM: Extracting dynamic time-delays to reconstruct multivariate data for improving attention-based LSTM industrial time series prediction models

Abstract Taking advantage of varying degrees of attention on specific features, attention-based long short-term memory (ALSTM) networks have made inroads into the industrial multivariate time series prediction sector recently. However, conventional ALSTM models usually employ static time-delays to constant select input/output pairings of multivariate data, which apparently ignores dynamics of transmission time between industrial process variables and degrades the prediction performance by incorrectly extracting process characteristics. In response to this problem, this paper proposes a novel approach to extracting dynamic time-delays to reconstruct (DTDR) multivariate data for an improved ALSTM prediction model. Therein, the temporal locations and spans of multivariate data are adaptively tailored to input/output pairings of the ALSTM network according to the dynamic time-delays. Specifically, the multivariate data can be accurately matched in temporal positions, and the data information in the original temporal spans with Status transfer time abnormal are replaced. Consequently, this prediction model not only appropriately utilizes dynamics between the predicting and correlated variables, but also makes better attentions on key features extracted from optimum data. Applied to industrial distillation and methanol production processes, the proposed method demonstrates the capability of significantly improving network training speeds as well as prediction accuracies in contrast to static time-delay based ALSTM and LSTM models, expecting even more applications.

[1]  Junyu Dong,et al.  Prediction of Sea Surface Temperature Using Long Short-Term Memory , 2017, IEEE Geoscience and Remote Sensing Letters.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Laurie M. Linnett,et al.  High resolution time delay estimation using sliding discrete Fourier transform , 2012, Digit. Signal Process..

[4]  Lijuan Sun,et al.  Water Quality Prediction Method Based on IGRA and LSTM , 2018, Water.

[5]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[6]  Tianrui Li,et al.  Multivariate time series forecasting via attention-based encoder-decoder framework , 2020, Neurocomputing.

[7]  S L Shah,et al.  Improved correlation analysis and visualization of industrial alarm data. , 2012, ISA transactions.

[8]  Bo Yang,et al.  A similarity elastic window based approach to process dynamic time delay analysis , 2017 .

[9]  Le Kang,et al.  Hybrid genetic algorithm method for efficient and robust evaluation of remaining useful life of supercapacitors , 2020 .

[10]  Pengcheng Zhang,et al.  Interpretable spatio-temporal attention LSTM model for flood forecasting , 2020, Neurocomputing.

[11]  Yongmao Xu,et al.  Neural networks based model predictive control of an industrial polypropylene process , 2002, Proceedings of the International Conference on Control Applications.

[12]  Jun Hu,et al.  A deep learning model to effectively capture mutation information in multivariate time series prediction , 2020, Knowl. Based Syst..

[13]  Rui Li,et al.  Runtime Quality Prediction for Web Services via Multivariate Long Short-Term Memory , 2019 .

[14]  Chenghui Zhang,et al.  Parameters Identification and Sensitive Characteristics Analysis for Lithium-Ion Batteries of Electric Vehicles , 2017 .

[15]  Jun Hu,et al.  Multistage attention network for multivariate time series prediction , 2020, Neurocomputing.

[16]  Y. X. Huang,et al.  Time dependent intrinsic correlation analysis of temperature and dissolved oxygen time series using empirical mode decomposition , 2013, 1401.4201.

[17]  Li-Chiu Chang,et al.  Explore a deep learning multi-output neural network for regional multi-step-ahead air quality forecasts , 2019, Journal of Cleaner Production.

[18]  Pan Zhang,et al.  A Multilevel Prediction Model of Carbon Efficiency Based on the Differential Evolution Algorithm for the Iron Ore Sintering Process , 2018, IEEE Transactions on Industrial Electronics.

[19]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[20]  Yongjian Wang,et al.  An Improved Bar-Shaped Sliding Window CNN Tailored to Industrial Process Historical Data with Applications in Chemical Operational Optimizations , 2019, Industrial & Engineering Chemistry Research.

[21]  Javier Gonzalez-Jimenez,et al.  Continuous chemical classification in uncontrolled environments with sliding windows , 2016 .

[22]  Duo Zhang,et al.  Use Long Short-Term Memory to Enhance Internet of Things for Combined Sewer Overflow Monitoring , 2018 .

[23]  Witold Pedrycz,et al.  The modeling of time series based on fuzzy information granules , 2014, Expert Syst. Appl..

[24]  Yeonguk Yu,et al.  Attention-LSTM-Attention Model for Speech Emotion Recognition and Analysis of IEMOCAP Database , 2020, Electronics.

[25]  Bo Yang,et al.  A novel dynamic timed fuzzy Petri nets modeling method with applications to industrial processes , 2018, Expert Syst. Appl..

[26]  Hongguang Li,et al.  Hybrid cycle reservoir with jumps for multivariate time series prediction: industrial application in oil drilling process , 2019, Measurement Science and Technology.

[27]  Yang Du,et al.  Short-Term Photovoltaic Power Forecasting Based on Long Short Term Memory Neural Network and Attention Mechanism , 2019, IEEE Access.

[28]  Qunxiong Zhu,et al.  Energy saving and prediction modeling of petrochemical industries: A novel ELM based on FAHP , 2017 .

[29]  Ling Yang,et al.  DSTP-RNN: a dual-stage two-phase attention-based recurrent neural networks for long-term and multivariate time series prediction , 2019, Expert Syst. Appl..

[30]  Marcelo H. Ang,et al.  Traffic light status detection using movement patterns of vehicles , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[31]  Yu Zhang,et al.  Prediction of Wind Turbine-Grid Interaction Based on a Principal Component Analysis-Long Short Term Memory Model , 2018, Energies.

[32]  Du Q. Huynh,et al.  A Location-Velocity-Temporal Attention LSTM Model for Pedestrian Trajectory Prediction , 2020, IEEE Access.

[33]  Rohitash Chandra,et al.  Identification of minimal timespan problem for recurrent neural networks with application to cyclone wind-intensity prediction , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[34]  Chuan Ding,et al.  Parallel Architecture of Convolutional Bi-Directional LSTM Neural Networks for Network-Wide Metro Ridership Prediction , 2019, IEEE Transactions on Intelligent Transportation Systems.

[35]  Bin Li,et al.  LSTM-Attention-Embedding Model-Based Day-Ahead Prediction of Photovoltaic Power Output Using Bayesian Optimization , 2019, IEEE Access.

[36]  Chen-hua Shen,et al.  Analysis of detrended time-lagged cross-correlation between two nonstationary time series , 2015 .

[37]  Xiaohui Liang,et al.  CTS-LSTM: LSTM-based neural networks for correlatedtime series prediction , 2020, Knowl. Based Syst..