A novel semi-supervised pre-training strategy for deep networks and its application for quality variable prediction in industrial processes

Abstract Deep learning-based soft sensor has been a hot topic for quality variable prediction in modern industrial processes. Feature representation with deep learning is the key step to build an accurate and reliable soft sensor model from massive process data. To deal with the limited labeled data and abundant unlabeled data, a semi-supervised pre-training strategy is proposed for deep learning network in this paper, which is based on semi-supervised stacked autoencoder (SS-SAE). For traditional deep networks like SAE, the pre-training procedure is unsupervised and may discard important information in the labeled data. Different from them, SS-SAE automatically adjusts the training strategy according to the given data type. For unlabeled data, it learns the shape of the input distribution layer by layer. While for labeled data, it additionally learns quality-related features with the guidance of quality information. The proposed method is validated on two refining industries of a debutanizer column and a hydrocracking process. The results show that SS-SAE can utilize both labeled and unlabeled data to extract quality-relevant features for soft sensor modeling, which is superior to multi-layer neural network, traditional SAE and DBN.

[1]  Sirish L. Shah,et al.  Inferential sensors for estimation of polymer quality parameters: Industrial application of a PLS-based soft sensor for a LDPE plant , 2006 .

[2]  Steven X. Ding,et al.  A New Soft-Sensor-Based Process Monitoring Scheme Incorporating Infrequent KPI Measurements , 2015, IEEE Transactions on Industrial Electronics.

[3]  Manabu Kano,et al.  Long-Term Industrial Applications of Inferential Control Based on Just-In-Time Soft-Sensors: Economical Impact and Challenges , 2013 .

[4]  Zhiqiang Ge,et al.  Weighted Linear Dynamic System for Feature Representation and Soft Sensor Application in Nonlinear Dynamic Industrial Processes , 2018, IEEE Transactions on Industrial Electronics.

[5]  Zhi-huan Song,et al.  Locally Weighted Kernel Principal Component Regression Model for Soft Sensing of Nonlinear Time-Variant Processes , 2014 .

[6]  Jiwen Lu,et al.  Single Sample Face Recognition via Learning Deep Supervised Autoencoders , 2015, IEEE Transactions on Information Forensics and Security.

[7]  Yue Cao,et al.  A Novel Sliding Window PCA-IPF Based Steady-State Detection Framework and Its Industrial Application , 2018, IEEE Access.

[8]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.

[9]  Biao Huang,et al.  A Bayesian framework for real‐time identification of locally weighted partial least squares , 2015 .

[10]  Zhiqiang Ge,et al.  Deep Learning of Semisupervised Process Data With Hierarchical Extreme Learning Machine and Soft Sensor Application , 2018, IEEE Transactions on Industrial Electronics.

[11]  Furong Gao,et al.  Review of Recent Research on Data-Based Process Monitoring , 2013 .

[12]  Robert P. Sheridan,et al.  Deep Neural Nets as a Method for Quantitative Structure-Activity Relationships , 2015, J. Chem. Inf. Model..

[13]  S. Ray,et al.  Nonlinear control of debutanizer column using profile position observer , 2009, Comput. Chem. Eng..

[14]  Weihua Gui,et al.  A Layer-Wise Data Augmentation Strategy for Deep Learning Networks and Its Soft Sensor Application in an Industrial Hydrocracking Process , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[16]  Per Julian Becker,et al.  A single events microkinetic model for hydrocracking of vacuum gas oil , 2017, Comput. Chem. Eng..

[17]  Zhiqiang Ge,et al.  Probabilistic latent variable regression model for process-quality monitoring , 2014 .

[18]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[19]  Luigi Fortuna,et al.  Soft sensors for product quality monitoring in debutanizer distillation columns , 2005 .

[20]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[21]  Weiming Shao,et al.  Adaptive soft sensor for quality prediction of chemical processes based on selective ensemble of local partial least squares models , 2015 .

[22]  Jun Yu,et al.  Multitask Autoencoder Model for Recovering Human Poses , 2018, IEEE Transactions on Industrial Electronics.

[23]  Zhiqiang Ge,et al.  A Probabilistic Just-in-Time Learning Framework for Soft Sensor Development With Missing Data , 2017, IEEE Transactions on Control Systems Technology.

[24]  Dexian Huang,et al.  Data-driven soft sensor development based on deep learning technique , 2014 .

[25]  S. Das,et al.  Non-spherical solid-non-Newtonian liquid fluidization and ANN modelling: Minimum fluidization velocity , 2018 .

[26]  Raymond Lau,et al.  Modeling the change in particle size distribution in a gas-solid fluidized bed due to particle attrition using a hybrid artificial neural network-genetic algorithm approach , 2016 .

[27]  Yi Liu,et al.  Development of soft-sensors for online quality prediction of sequential-reactor-multi-grade industrial processes , 2013 .

[28]  Biao Huang,et al.  Deep Learning-Based Feature Representation and Its Application for Soft Sensor Modeling With Variable-Wise Weighted SAE , 2018, IEEE Transactions on Industrial Informatics.

[29]  Zhiqiang Ge,et al.  Quality variable prediction for chemical processes based on semisupervised Dirichlet process mixture of Gaussians , 2019, Chemical Engineering Science.

[30]  Weihua Gui,et al.  Temperature prediction for roller kiln based on hybrid first-principle model and data-driven MW-DLWKPCR model. , 2020, ISA transactions.

[31]  Xiaofeng Yuan,et al.  A two‐layer ensemble learning framework for data‐driven soft sensor of the diesel attributes in an industrial hydrocracking process , 2019, Journal of Chemometrics.

[32]  Manabu Kano,et al.  Data-based process monitoring, process control, and quality improvement: Recent developments and applications in steel industry , 2008, Comput. Chem. Eng..

[33]  Davide Fissore,et al.  Design and validation of an innovative soft-sensor for pharmaceuticals freeze-drying monitoring , 2011 .

[34]  Amiya K. Jana,et al.  Nonlinear state estimation and control of a refinery debutanizer column , 2009, Comput. Chem. Eng..

[35]  Bogdan Gabrys,et al.  Data-driven Soft Sensors in the process industry , 2009, Comput. Chem. Eng..

[36]  Weihua Gui,et al.  Deep quality-related feature extraction for soft sensing modeling: A deep learning approach with hybrid VW-SAE , 2020, Neurocomputing.

[37]  Weihua Gui,et al.  A novel deep learning based fault diagnosis approach for chemical process with extended deep belief network. , 2019, ISA transactions.

[38]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[39]  Zhiqiang Ge,et al.  Robust supervised probabilistic principal component analysis model for soft sensing of key process variables , 2015 .

[40]  Honglai Liu,et al.  Machine learning models for solvent effects on electric double layer capacitance , 2019, Chemical Engineering Science.

[41]  Yoshua Bengio,et al.  What regularized auto-encoders learn from the data-generating distribution , 2012, J. Mach. Learn. Res..

[42]  Hiromasa Kaneko,et al.  Application of Online Support Vector Regression for Soft Sensors , 2014 .

[43]  Won Bo Lee,et al.  Toxic gas release modeling for real-time analysis using variational autoencoder with convolutional neural networks , 2018 .

[44]  Biao Huang,et al.  Dynamic Modelling and Predictive Control in Solid Oxide Fuel Cells: First Principle and Data-Based Approaches: Huang/Dynamic Modelling and Predictive Control in Solid Oxide Fuel Cells: First Principle and Data-Based Approaches , 2013 .

[45]  Zhiqiang Ge,et al.  Semisupervised JITL Framework for Nonlinear Industrial Soft Sensing Based on Locally Semisupervised Weighted PCR , 2017, IEEE Transactions on Industrial Informatics.

[46]  Chunhui Zhao,et al.  A new soft-sensor algorithm with concurrent consideration of slowness and quality interpretation for dynamic chemical process , 2019, Chemical Engineering Science.

[47]  Ning Chen,et al.  Temperature Prediction Model for Roller Kiln by ALD-Based Double Locally Weighted Kernel Principal Component Regression , 2018, IEEE Transactions on Instrumentation and Measurement.

[48]  Yoshua Bengio,et al.  Practical Recommendations for Gradient-Based Training of Deep Architectures , 2012, Neural Networks: Tricks of the Trade.