An empirical evaluation of attention-based multi-head models for improved turbofan engine remaining useful life prediction

A single unit (head) is the conventional input feature extractor in deep learning architectures trained on multivariate time series signals. The importance of the fixed-dimensional vector representation generated by the single-head network has been demonstrated for industrial machinery condition monitoring and predictive maintenance. However, processing heterogeneous sensor signals with a single-head may result in a model that cannot explicitly account for the diversity in time-varying multivariate inputs. This work extends the conventional single-head deep learning models to a more robust form by developing context-specific heads to independently capture the inherent pattern in each sensor reading. Using the turbofan aircraft engine benchmark dataset (CMAPSS), an extensive experiment is performed to verify the effectiveness and benefits of multi-head multilayer perceptron, recurrent networks, convolution network, the transformerstyle stand-alone attention network, and their variants for remaining useful life estimation. Moreover, the effect of different attention mechanisms on the multi-head models is also evaluated. In addition, each architecture's relative advantage and computational overhead are analyzed. Results show that utilizing the attention layer is task-sensitive and model dependent, as it does not provide consistent improvement across the models investigated. The best model is further compared with five state-of-the-art models, and the comparison shows that a relatively simple multi-head architecture performs better than the state-of-the-art models. The results presented in this study demonstrate the importance of multi-head models and attention mechanisms to improved understanding of the remaining useful life of industrial assets. Keyword: CMAPSS; deep learning, remaining useful life, attention mechanism, predictive maintenance. * Corresponding author. E-mail address: lxg@zju.edu.cn (X. Liu).

[1]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[2]  Ajith Abraham,et al.  A novel joint histogram equalization based image contrast enhancement , 2019, J. King Saud Univ. Comput. Inf. Sci..

[3]  Xinggao Liu,et al.  Mutation grey wolf elite PSO balanced XGBoost for radar emitter individual identification based on measured signals , 2020 .

[4]  Enrique Onieva,et al.  Multi-head CNN-RNN for multi-time series anomaly detection: An industrial case study , 2019, Neurocomputing.

[5]  Yangyang Wang,et al.  Long short-term memory neural network with weight amplification and its application into gear remaining useful life prediction , 2020, Eng. Appl. Artif. Intell..

[6]  Xinggao Liu,et al.  A novel effective diagnosis model based on optimized least squares support machine for gene microarray , 2018, Appl. Soft Comput..

[7]  Qian Liu,et al.  Gated recurrent unit based recurrent neural network for remaining useful life prediction of nonlinear deterioration process , 2019, Reliab. Eng. Syst. Saf..

[8]  Zhipeng Xu,et al.  A robust reliability prediction method using Weighted Least Square Support Vector Machine equipped with Chaos Modified Particle Swarm Optimization and Online Correcting Strategy , 2019, Appl. Soft Comput..

[9]  Moncef Gabbouj,et al.  Real-Time Fault Detection and Identification for MMC Using 1-D Convolutional Neural Networks , 2019, IEEE Transactions on Industrial Electronics.

[10]  Xinggao Liu,et al.  A novel shearer cutting pattern recognition model with chaotic gravitational search optimization , 2019, Measurement.

[11]  Ching-Chang Wong,et al.  Evolutional RBFNs image model describing-based segmentation system designs , 2018, Neurocomputing.

[12]  Hsuan-Ming Feng,et al.  Evolutional RBFNs prediction systems generation in the applications of financial time series data , 2011, Expert Syst. Appl..

[13]  M. A. Djeziri,et al.  Data-driven approach augmented in simulation for robust fault prognosis , 2019, Eng. Appl. Artif. Intell..

[14]  Patrick Siarry,et al.  A novel disturbance rejection factor based stable direct adaptive fuzzy control strategy for a class of nonlinear systems , 2020 .

[15]  Yong-kuo Liu,et al.  Support vector ensemble for incipient fault diagnosis in nuclear plant components , 2018, Nuclear Engineering and Technology.

[16]  W. Wang,et al.  A data-model-fusion prognostic framework for dynamic system state forecasting , 2012, Eng. Appl. Artif. Intell..

[17]  Abhinav Saxena,et al.  Damage propagation modeling for aircraft engine run-to-failure simulation , 2008, 2008 International Conference on Prognostics and Health Management.

[18]  Rongrong Ying,et al.  Remaining useful life estimation with multiple local similarities , 2020, Eng. Appl. Artif. Intell..

[19]  Wenhai Wang,et al.  A robust cutting pattern recognition method for shearer based on Least Square Support Vector Machine equipped with Chaos Modified Particle Swarm Optimization and Online Correcting Strategy. , 2019, ISA transactions.

[20]  Shuo Xu,et al.  Remaining Useful Life Estimation Using Deep Convolutional Generative Adversarial Networks Based on an Autoencoder Scheme , 2020, Comput. Intell. Neurosci..

[21]  Bowen Zhou,et al.  A Structured Self-attentive Sentence Embedding , 2017, ICLR.

[22]  Lu Lv,et al.  A novel intrusion detection system based on an optimal hybrid kernel extreme learning machine , 2020, Knowl. Based Syst..

[23]  Robert B. Randall,et al.  Vibration-based updating of wear prediction for spur gears , 2019, Wear.

[24]  Yong-kuo Liu,et al.  PWR heat exchanger tube defects: Trends, signatures and diagnostic techniques , 2019, Progress in Nuclear Energy.

[25]  Yong-kuo Liu,et al.  SVR optimization with soft computing algorithms for incipient SGTR diagnosis , 2018, Annals of Nuclear Energy.

[26]  Alex Graves,et al.  Neural Turing Machines , 2014, ArXiv.

[27]  Youxian Sun,et al.  A novel fault diagnosis method based on optimal relevance vector machine , 2017, Neurocomputing.

[28]  Yong Li,et al.  A generalized remaining useful life prediction method for complex systems based on composite health indicator , 2021, Reliab. Eng. Syst. Saf..

[29]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[30]  Patrick Siarry,et al.  A robust FLIR target detection employing an auto-convergent pulse coupled neural network , 2019, Remote Sensing Letters.

[31]  Zhong Cheng,et al.  Optimal online soft sensor for product quality monitoring in propylene polymerization process , 2015, Neurocomputing.

[32]  Xiang Li,et al.  Remaining useful life estimation in prognostics using deep convolution neural networks , 2018, Reliab. Eng. Syst. Saf..

[33]  Patrick Siarry,et al.  Multi-objective design of optimal higher order sliding mode control for robust tracking of 2-DoF helicopter system based on metaheuristics , 2019, Aerospace Science and Technology.

[34]  Chunhua Yang,et al.  Causal augmented ConvNet: A temporal memory dilated convolution model for long-sequence time series prediction. , 2021, ISA transactions.

[35]  Yuehui Chen,et al.  Cyber Security And The Evolution Of Intrusion Detection Systems , 2005 .

[36]  Xinggao Liu,et al.  Squeeze excitation densely connected residual convolutional networks for specific emitter identification based on measured signals , 2020, Measurement Science and Technology.

[37]  Mohamed Benbouzid,et al.  Aircraft engines Remaining Useful Life prediction with an adaptive denoising online sequential Extreme Learning Machine , 2020, Eng. Appl. Artif. Intell..

[38]  Zhongxiao Peng,et al.  Use of cyclostationary properties of vibration signals to identify gear wear mechanisms and track wear evolution , 2021 .

[39]  Wennian Yu,et al.  Remaining useful life estimation using a bidirectional recurrent neural network based autoencoder scheme , 2019, Mechanical Systems and Signal Processing.

[40]  Yong-kuo Liu,et al.  A new perspective towards the development of robust data-driven intrusion detection for industrial control systems , 2020, Nuclear Engineering and Technology.

[41]  Houxiang Zhang,et al.  Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture , 2019, Reliab. Eng. Syst. Saf..

[42]  Yong-kuo Liu,et al.  Acoustic Signal-based Leak Size Estimation for Electric Valves Using Deep Belief Network , 2019, 2019 IEEE 5th International Conference on Computer and Communications (ICCC).

[43]  Youxian Sun,et al.  Application of Takagi–Sugeno fuzzy model optimized with an improved Free Search algorithm to industrial polypropylene melt index prediction , 2017 .