Deep Learning With Spatiotemporal Attention-Based LSTM for Industrial Soft Sensor Model Development

Industrial process data are naturally complex time series with high nonlinearities and dynamics. To model nonlinear dynamic processes, a long short-term memory (LSTM) network is very suitable for soft sensor model development. However, the original LSTM does not consider variable and sample relevance for quality prediction. In order to overcome this problem, a spatiotemporal attention-based LSTM network is proposed for soft sensor modeling, which can, not only identify important input variables that are related to the quality variable at each time step, but also adaptively discover quality-related hidden states across all time steps. By taking the spatiotemporal quality-relevant interactions into consideration, the prediction performance can be improved for the soft sensor model. The effectiveness and flexibility of the proposed model is demonstrated on an industrial hydrocracking process to predict the initial boiling points of heavy naphtha and aviation kerosene.

[1]  Dexian Huang,et al.  Data-driven soft sensor development based on deep learning technique , 2014 .

[2]  Jürgen Schmidhuber,et al.  Applying LSTM to Time Series Predictable through Time-Window Approaches , 2000, ICANN.

[3]  Alex Graves,et al.  Supervised Sequence Labelling with Recurrent Neural Networks , 2012, Studies in Computational Intelligence.

[4]  Zhiqiang Ge,et al.  Deep Learning of Semisupervised Process Data With Hierarchical Extreme Learning Machine and Soft Sensor Application , 2018, IEEE Transactions on Industrial Electronics.

[5]  Ning Chen,et al.  Temperature Prediction Model for Roller Kiln by ALD-Based Double Locally Weighted Kernel Principal Component Regression , 2018, IEEE Transactions on Instrumentation and Measurement.

[6]  Geoffrey E. Hinton.  Learning multiple layers of representation , 2007, Trends in Cognitive Sciences.

[7]  Xiaofeng Yuan,et al.  A spatial-temporal LWPLS for adaptive soft sensor modeling and its application for an industrial hydrocracking process , 2020 .

[8]  Weihua Gui,et al.  Deep learning for quality prediction of nonlinear dynamic processes with variable attention‐based long short‐term memory network , 2020 .

[9]  Navdeep Jaitly,et al.  Hybrid speech recognition with Deep Bidirectional LSTM , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.

[10]  Zhiqiang Ge,et al.  Weighted Linear Dynamic System for Feature Representation and Soft Sensor Application in Nonlinear Dynamic Industrial Processes , 2018, IEEE Transactions on Industrial Electronics.

[11]  Bhuvana Ramabhadran,et al.  Language modeling with highway LSTM , 2017, 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).

[12]  Markku Niemela,et al.  Soft-Sensor-Based Flow Rate and Specific Energy Estimation of Industrial Variable-Speed-Driven Twin Rotary Screw Compressor , 2016, IEEE Transactions on Industrial Electronics.

[13]  Yalin Wang,et al.  Stacked Enhanced Auto-Encoder for Data-Driven Soft Sensing of Quality Variable , 2020, IEEE Transactions on Instrumentation and Measurement.

[14]  Weihua Gui,et al.  Temperature prediction for roller kiln based on hybrid first-principle model and data-driven MW-DLWKPCR model. , 2020, ISA transactions.

[15]  S. Graziani,et al.  A deep learning based soft sensor for a sour water stripping plant , 2017, 2017 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[16]  Weihua Gui,et al.  A novel deep learning based fault diagnosis approach for chemical process with extended deep belief network. , 2019, ISA transactions.

[17]  Zhiqiang Ge,et al.  Probabilistic learning of partial least squares regression model: Theory and industrial applications , 2016 .

[18]  Biao Huang,et al.  Design of inferential sensors in the process industry: A review of Bayesian methods , 2013 .

[19]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[20]  S. Billings,et al.  A prediction-error and stepwise-regression estimation algorithm for non-linear systems , 1986 .

[21]  Weihua Gui,et al.  A novel semi-supervised pre-training strategy for deep networks and its application for quality variable prediction in industrial processes , 2020 .

[22]  Lin Li,et al.  Nonlinear Dynamic Soft Sensor Modeling With Supervised Long Short-Term Memory Network , 2020, IEEE Transactions on Industrial Informatics.

[23]  Junghui Chen,et al.  Deep Learning of Complex Batch Process Data and Its Application on Quality Prediction , 2020, IEEE Transactions on Industrial Informatics.

[24]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[25]  Biao Huang,et al.  Hierarchical Quality-Relevant Feature Representation for Soft Sensor Modeling: A Novel Deep Learning Strategy , 2020, IEEE Transactions on Industrial Informatics.

[26]  Xiaofeng Yuan,et al.  A two‐layer ensemble learning framework for data‐driven soft sensor of the diesel attributes in an industrial hydrocracking process , 2019, Journal of Chemometrics.

[27]  S. Billings,et al.  Correlation based model validity tests for non-linear models , 1986 .

[28]  Zhiqiang Ge,et al.  Semisupervised JITL Framework for Nonlinear Industrial Soft Sensing Based on Locally Semisupervised Weighted PCR , 2017, IEEE Transactions on Industrial Informatics.

[29]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[30]  Weihua Gui,et al.  A Deep Supervised Learning Framework for Data-Driven Soft Sensor Modeling of Industrial Processes , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Weiming Shao,et al.  Adaptive soft sensor for quality prediction of chemical processes based on selective ensemble of local partial least squares models , 2015 .

[32]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[33]  Xiaofeng Yuan,et al.  Multi‐similarity measurement driven ensemble just‐in‐time learning for soft sensing of industrial processes , 2018 .

[34]  Weihua Gui,et al.  Stacked isomorphic autoencoder based soft analyzer and its application to sulfur recovery unit , 2020, Inf. Sci..

[35]  Weihua Gui,et al.  A Layer-Wise Data Augmentation Strategy for Deep Learning Networks and Its Soft Sensor Application in an Industrial Hydrocracking Process , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[36]  Zhiqiang Ge,et al.  Mixture Bayesian Regularization of PCR Model and Soft Sensing Application , 2015, IEEE Transactions on Industrial Electronics.

[37]  Biao Huang,et al.  Deep Learning-Based Feature Representation and Its Application for Soft Sensor Modeling With Variable-Wise Weighted SAE , 2018, IEEE Transactions on Industrial Informatics.

[38]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[39]  Nasser Mohamed Ramli,et al.  Composition Prediction of a Debutanizer Column using Equation Based Artificial Neural Network Model , 2014, Neurocomputing.

[40]  Xiaofeng Yuan,et al.  A Comparative Study of Adaptive Soft Sensors for Quality Prediction in an Industrial Refining Hydrocracking Process , 2018, 2018 IEEE 7th Data Driven Control and Learning Systems Conference (DDCLS).

[41]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[42]  Deoki N. Saraf,et al.  Design of neural networks using genetic algorithm for on-line property estimation of crude fractionator products , 2006, Comput. Chem. Eng..

[43]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[44]  Di Tang,et al.  A Data-Driven Soft Sensor Modeling Method Based on Deep Learning and its Application , 2017, IEEE Transactions on Industrial Electronics.

[45]  Zhiqiang Ge,et al.  A Probabilistic Just-in-Time Learning Framework for Soft Sensor Development With Missing Data , 2017, IEEE Transactions on Control Systems Technology.

[46]  Weihua Gui,et al.  Deep quality-related feature extraction for soft sensing modeling: A deep learning approach with hybrid VW-SAE , 2020, Neurocomputing.