Ensemble of optimized echo state networks for remaining useful life prediction

Abstract The use of Echo State Networks (ESNs) for the prediction of the Remaining Useful Life (RUL) of industrial components, i.e. the time left before the equipment will stop fulfilling its functions, is attractive because of their capability of handling the system dynamic behavior, the measurement noise, and the stochasticity of the degradation process. In particular, in this paper we originally resort to an ensemble of ESNs, for enhancing the performances of individual ESNs and providing also an estimation of the uncertainty affecting the RUL prediction. The main methodological novelties in our use of ESNs for RUL prediction are: i) the use of the individual ESN memory capacity within the dynamic procedure for aggregating of the ESNs outcomes; ii) the use of an additional ESN for estimating the RUL uncertainty, within the Mean Variance Estimation (MVE) approach. With these novelties, the developed approach outperforms a static ensemble and a standard MVE approach for uncertainty estimation in tests performed on a synthetic and two industrial datasets.

[1]  Amaury Lendasse,et al.  Adding reliability to ELM forecasts by confidence intervals , 2017, Neurocomputing.

[2]  Marios M. Polycarpou,et al.  High-order neural network structures for identification of dynamical systems , 1995, IEEE Trans. Neural Networks.

[3]  Enrico Zio,et al.  A belief function theory based approach to combining different representation of uncertainty in prognostics , 2015, Inf. Sci..

[4]  Piero Baraldi,et al.  Ensemble of bootstrapped models for the prediction of the remaining useful life of a creeping turbine blade , 2012, 2012 IEEE Conference on Prognostics and Health Management.

[5]  A. Weigend,et al.  Estimating the mean and variance of the target probability distribution , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[6]  Enrico Zio,et al.  NSGA-II-trained neural network approach to the estimation of prediction intervals of scale deposition rate in oil & gas equipment , 2013, Expert Syst. Appl..

[7]  Irena Koprinska,et al.  Prediction intervals for electricity load forecasting using neural networks , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[8]  Oliver Kramer,et al.  Wind energy prediction and monitoring with neural computation , 2013, Neurocomputing.

[9]  Istvan Szunyogh,et al.  A local ensemble Kalman filter for atmospheric data assimilation , 2004 .

[10]  R. Gouriveau,et al.  Fuel Cells prognostics using echo state network , 2013, IECON 2013 - 39th Annual Conference of the IEEE Industrial Electronics Society.

[11]  Hamid Reza Karimi,et al.  Model approximation for two-dimensional Markovian jump systems with state-delays and imperfect mode information , 2015, Multidimens. Syst. Signal Process..

[12]  Kai Goebel,et al.  Comparison of prognostic algorithms for estimating remaining useful life of batteries , 2009 .

[13]  Enrico Zio,et al.  Model-based and data-driven prognostics under different available information , 2013 .

[14]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[15]  Amir F. Atiya,et al.  Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances , 2011, IEEE Transactions on Neural Networks.

[16]  Liping Xie,et al.  Direct interval forecasting of wind speed using radial basis function neural networks in a multi-objective optimization framework , 2016, Neurocomputing.

[17]  Istvan Szunyogh,et al.  A Local Ensemble Kalman Filter for Atmospheric Data Assimilation , 2002 .

[18]  Krishna R. Pattipati,et al.  Model-Based Prognostic Techniques Applied to a Suspension System , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[19]  Richard D. Braatz,et al.  Fault Detection and Diagnosis in Industrial Systems , 2001 .

[20]  K. Pipe.  Practical prognostics for Condition Based Maintenance , 2008, 2008 International Conference on Prognostics and Health Management.

[21]  R. Polikar,et al.  Ensemble based systems in decision making , 2006, IEEE Circuits and Systems Magazine.

[22]  Nando de Freitas,et al.  Sequential Monte Carlo Methods in Practice , 2001, Statistics for Engineering and Information Science.

[23]  Saeid Nahavandi,et al.  Improving the Quality of Prediction Intervals Through Optimal Aggregation , 2015, IEEE Transactions on Industrial Electronics.

[24]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[25]  Sankalita Saha,et al.  Metrics for Offline Evaluation of Prognostic Performance , 2021, International Journal of Prognostics and Health Management.

[26]  Xenofon D. Koutsoukos,et al.  A Comprehensive Diagnosis Methodology for Complex Hybrid Systems: A Case Study on Spacecraft Power Distribution Systems , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[27]  David R. Jacques,et al.  Impact of Prognostic Uncertainty in System Health Monitoring , 2015 .

[28]  B. Samanta,et al.  ARTIFICIAL NEURAL NETWORK BASED FAULT DIAGNOSTICS OF ROLLING ELEMENT BEARINGS USING TIME-DOMAIN FEATURES , 2003 .

[29]  Peter W. Tse,et al.  Prediction of Machine Deterioration Using Vibration Based Fault Trends and Recurrent Neural Networks , 1999 .

[30]  Shu-Xian Lun,et al.  A novel model of leaky integrator echo state network for time-series prediction , 2015, Neurocomputing.

[31]  Yang Wang,et al.  Repairing the crossover rate in adaptive differential evolution , 2014, Appl. Soft Comput..

[32]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[33]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[34]  Steve Temple,et al.  Spiking neural networkS on Spinnaker By , 2010 .

[35]  Piero P. Bonissone,et al.  Fast meta-models for local fusion of multiple predictive models , 2011, Appl. Soft Comput..

[36]  J. T. Hwang,et al.  Prediction Intervals for Artificial Neural Networks , 1997 .

[37]  Jinde Cao,et al.  Remaining useful life estimation using an inverse Gaussian degradation model , 2016, Neurocomputing.

[38]  H. Akaike.  Fitting autoregressive models for prediction , 1969 .

[39]  Abhinav Saxena,et al.  Damage propagation modeling for aircraft engine run-to-failure simulation , 2008, 2008 International Conference on Prognostics and Health Management.

[40]  Enrico Zio,et al.  A comparison between extreme learning machine and artificial neural network for remaining useful life prediction , 2016, 2016 Prognostics and System Health Management Conference (PHM-Chengdu).

[41]  John A. Bullinaria,et al.  Neural network ensembles for time series forecasting , 2009, GECCO '09.

[42]  L. Peel,et al.  Data driven prognostics using a Kalman filter ensemble of neural network models , 2008, 2008 International Conference on Prognostics and Health Management.

[43]  Rastko R. Selmic,et al.  Wireless Sensor Network Modeling Using Modified Recurrent Neural Networks: Application to Fault Detection , 2008, IEEE Transactions on Instrumentation and Measurement.

[44]  Wen Gao,et al.  Hierarchical Ensemble of Global and Local Classifiers for Face Recognition , 2009, IEEE Trans. Image Process..

[45]  Zhigang Zeng,et al.  Ensembles of echo state networks for time series prediction , 2013, 2013 Sixth International Conference on Advanced Computational Intelligence (ICACI).

[46]  Enrico Zio,et al.  RECURRENT NEURAL NETWORKS FOR DYNAMIC RELIABILITY ANALYSIS , 2007 .

[47]  Takashi Hiyama,et al.  Predicting remaining useful life of rotating machinery based artificial neural network , 2010, Comput. Math. Appl..

[48]  ENRICO ZIO,et al.  A Fuzzy Similarity-Based Method for Failure Detection and Recovery Time Estimation , 2010 .

[49]  Enrico Zio,et al.  Ensemble-approaches for clustering health status of oil sand pumps , 2012, Expert Syst. Appl..

[50]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[51]  Enrico Zio,et al.  Echo State Network for the Remaining Useful Life Prediction of a Turbofan Engine , 2016, PHM Society European Conference.

[52]  F.O. Heimes,et al.  Recurrent neural networks for remaining useful life estimation , 2008, 2008 International Conference on Prognostics and Health Management.

[53]  M.G. Pecht,et al.  Prognostics and health management of electronics , 2008, IEEE Transactions on Components and Packaging Technologies.

[54]  Wilson Wang,et al.  An evolving neuro-fuzzy technique for system state forecasting , 2012, Neurocomputing.

[55]  Joseph Mathew,et al.  A review on prognostic techniques for non-stationary and non-linear rotating systems , 2015 .

[56]  Enrico Zio,et al.  Quantifying uncertainties in the estimation of safety parameters by using bootstrapped artificial neural networks , 2008 .

[57]  Mark Schwabacher,et al.  A Survey of Data -Driven Prognostics , 2005 .

[58]  Simon J. Godsill,et al.  On sequential simulation-based methods for Bayesian filtering , 1998 .

[59]  Chao Hu,et al.  Ensemble of data-driven prognostic algorithms for robust prediction of remaining useful life , 2011, 2011 IEEE Conference on Prognostics and Health Management.

[60]  Jonathan S. Litt,et al.  User's Guide for the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) , 2007 .

[61]  Isis Didier Lins,et al.  Predictive maintenance policy for oil well equipment in case of scaling through support vector machines , 2011 .

[62]  Hamid Reza Karimi,et al.  Filtering design for two-dimensional Markovian jump systems with state-delays and deficient mode information , 2014, Inf. Sci..

[63]  Enrico Zio,et al.  Investigation of uncertainty treatment capability of model-based and data-driven prognostic methods using simulated data , 2013, Reliab. Eng. Syst. Saf..

[64]  Kwok-Leung Tsui,et al.  An ensemble model for predicting the remaining useful performance of lithium-ion batteries , 2013, Microelectron. Reliab..

[65]  Enrico Zio,et al.  Predicting time series of railway speed restrictions with time-dependent machine learning techniques , 2013, Expert Syst. Appl..

[66]  Enrico Zio,et al.  An Interval-Valued Neural Network Approach for Uncertainty Quantification in Short-Term Wind Speed Prediction , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[67]  Abbas Khosravi,et al.  Particle swarm optimization for construction of neural network-based prediction intervals , 2014, Neurocomputing.

[68]  Amir F. Atiya,et al.  Lower Upper Bound Estimation Method for Construction of Neural Network-Based Prediction Intervals , 2011, IEEE Transactions on Neural Networks.

[69]  Xiangji Huang,et al.  Boosting Prediction Accuracy on Imbalanced Datasets with SVM Ensembles , 2006, PAKDD.

[70]  Yu Peng,et al.  A modified echo state network based remaining useful life estimation approach , 2012, 2012 IEEE Conference on Prognostics and Health Management.

[71]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[72]  Bijaya K. Panigrahi,et al.  Point and prediction interval estimation for electricity markets with machine learning techniques and wavelet transforms , 2013, Neurocomputing.

[73]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[74]  S. Roberts,et al.  Confidence Intervals and Prediction Intervals for Feed-Forward Neural Networks , 2001 .

[75]  Gwo-Hshiung Tzeng,et al.  Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS , 2004, Eur. J. Oper. Res..

[76]  Sankalita Saha,et al.  Evaluating algorithm performance metrics tailored for prognostics , 2009, 2009 IEEE Aerospace conference.

[77]  Enrico Zio,et al.  Ensemble neural network-based particle filtering for prognostics , 2013 .

[78]  Enrico Zio,et al.  Classifier-ensemble incremental-learning procedure for nuclear transient identification at different operational conditions , 2011, Reliab. Eng. Syst. Saf..

[79]  Ching-Lai Hwang,et al.  Fuzzy Multiple Attribute Decision Making - Methods and Applications , 1992, Lecture Notes in Economics and Mathematical Systems.

[80]  Enrico Zio,et al.  Hierarchical k-nearest neighbours classification and binary differential evolution for fault diagnostics of automotive bearings operating under variable conditions , 2016, Eng. Appl. Artif. Intell..

[81]  Jun Cheng,et al.  Finite-time boundedness of state estimation for neural networks with time-varying delays , 2014, Neurocomputing.

[82]  Enrico Zio,et al.  Availability Model of a PHM-Equipped Component , 2017, IEEE Transactions on Reliability.

[83]  Hong Zhao,et al.  A Local Field Correlated and Monte Carlo Based Shallow Neural Network Model for Nonlinear Time Series Prediction , 2016, EAI Endorsed Trans. Scalable Inf. Syst..

[84]  Yaochu Jin,et al.  Evolutionary multi-objective generation of recurrent neural network ensembles for time series prediction , 2014, Neurocomputing.

[85]  E. Zio,et al.  A study of the bootstrap method for estimating the accuracy of artificial neural networks in predicting nuclear transient processes , 2006, IEEE Transactions on Nuclear Science.

[86]  R.R. Selmic,et al.  Wireless Sensor Network Modeling Using Modified Recurrent Neural Networks: Application to Fault Detection , 2008, 2007 IEEE International Conference on Networking, Sensing and Control.

[87]  Brigitte Chebel-Morello,et al.  Linear feature selection and classification using PNN and SFAM neural networks for a nearly online diagnosis of bearing naturally progressing degradations , 2015, Eng. Appl. Artif. Intell..

[88]  Junghui Chen,et al.  Ensemble local kernel learning for online prediction of distributed product outputs in chemical processes , 2015 .

[89]  Hong Qiao,et al.  A comparative study of two modeling approaches in neural networks , 2004, Neural Networks.

[90]  G. Kitagawa.  Non-Gaussian State—Space Modeling of Nonstationary Time Series , 1987 .

[91]  Hubert Cardot,et al.  A new boosting algorithm for improved time-series forecasting with recurrent neural networks , 2008, Inf. Fusion.