Performance Benchmarking and Analysis of Prognostic Methods for CMAPSS Datasets

Six years and more than seventy publications later this paper looks back and analyzes the development of prognostic algorithms using C-MAPSS datasets generated and disseminated by the prognostic center of excellence at NASA Ames Research Center. Among those datasets are five run-to-failure CMAPSS datasets that have been popular due to various characteristics applicable to prognostics. The C-MAPSS datasets pose several challenges that are inherent to general prognostics applications. In particular, management of high variability due to sensor noise, effects of operating conditions, and presence of multiple simultaneous fault modes are some factors that have great impact on the generalization capabilities of prognostics algorithms. More than seventy publications have used the C-MAPSS datasets for developing data driven prognostic algorithms. However, in the absence of performance benchmarking results and due to common misunderstandings in interpreting the relationships between these datasets, it has been difficult for the users to suitably compare their results. In addition to identifying differentiating characteristics in these datasets, this paper also provides performance results for the PHM’08 data challenge wining entries to serve as performance baseline. This paper summarizes various prognostic modeling efforts that used C-MAPSS datasets and provides guidelines and references to further usage of these datasets in a manner that allows clear and consistent comparison between different approaches.

[1]  Emmanuel Ramasso,et al.  Investigating computational geometry for failure prognostics , 2014, International Journal of Prognostics and Health Management.

[2]  Manzar Abbas.  System-level health assessment of complex engineered processes , 2010 .

[3]  Tony R. Martinez,et al.  Turning Bayesian model averaging into Bayesian model combination , 2011, The 2011 International Joint Conference on Neural Networks.

[4]  Ludmila I. Kuncheva,et al.  Classifier Ensembles for Changing Environments , 2004, Multiple Classifier Systems.

[5]  Hatem M. Elattar,et al.  Evaluation of Neural Networks in the Subject of Prognostics As Compared To Linear Regression Model , 2010 .

[6]  David P. Williams,et al.  Classification with imperfect labels for fault prediction , 2011, KDD4Service '11.

[7]  Noureddine Zerhouni,et al.  Reducing arbitrary choices in model building for prognostics: An approach by applying parsimony principle on an evolving neuro-fuzzy system , 2011, Microelectron. Reliab..

[8]  F.O. Heimes,et al.  Recurrent neural networks for remaining useful life estimation , 2008, 2008 International Conference on Prognostics and Health Management.

[9]  James C. Bezdek,et al.  Decision templates for multiple classifier fusion: an experimental comparison , 2001, Pattern Recognit..

[10]  Jamie B. Coble,et al.  Applying the General Path Model to Estimation of Remaining Useful Life , 2011, International Journal of Prognostics and Health Management.

[11]  Jianjun Shi,et al.  A Data-Level Fusion Model for Developing Composite Health Indices for Degradation Modeling and Prognostic Analysis , 2013, IEEE Transactions on Automation Science and Engineering.

[12]  Michael Pecht,et al.  Application of a state space modeling technique to system prognostics based on a health index for condition-based maintenance , 2012 .

[13]  Simon Haykin,et al.  A Signal Processing Framework Based on Dynamic Neural Networks with Application to Problems in Adaptation, Filtering and Classification , 2001 .

[14]  Chao Hu,et al.  Ensemble of data-driven prognostic algorithms for robust prediction of remaining useful life , 2011, 2011 IEEE Conference on Prognostics and Health Management.

[15]  Emmanuel Ramasso,et al.  Contribution of belief functions to hidden markov models with an application to fault diagnosis , 2009, 2009 IEEE International Workshop on Machine Learning for Signal Processing.

[16]  Noureddine Zerhouni,et al.  E2GK-pro : An evidential evolving multimodeling approach for systems behavior prediction. , 2011 .

[17]  Ming Jian Zuo,et al.  An integrated framework for online diagnostic and prognostic health monitoring using a multistate deterioration process , 2014, Reliab. Eng. Syst. Saf..

[18]  Rafael Gouriveau,et al.  Prognostics in switching systems: Evidential markovian classification of real-time neuro-fuzzy predictions , 2010, 2010 Prognostics and System Health Management Conference.

[19]  Peter Willett,et al.  Comparison of data reduction techniques based on SVM classifier and SVR performance , 2011, Optical Engineering + Applications.

[20]  Xiaobin Li,et al.  Fault Prognostic Based on Hybrid Method of State Judgment and Regression , 2013 .

[21]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[22]  Noureddine Zerhouni,et al.  Novel failure prognostics approach with dynamic thresholds for machine degradation , 2013, IECON 2013 - 39th Annual Conference of the IEEE Industrial Electronics Society.

[23]  Jamie B. Coble,et al.  Merging Data Sources to Predict Remaining Useful Life – An Automated Method to Identify Prognostic Parameters , 2010 .

[24]  Tianyi Wang,et al.  Trajectory Similarity Based Prediction for Remaining Useful Life Estimation , 2010 .

[25]  Jonathan S. Litt,et al.  User's Guide for the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) , 2007 .

[26]  Abhinav Saxena,et al.  Damage propagation modeling for aircraft engine run-to-failure simulation , 2008, 2008 International Conference on Prognostics and Health Management.

[27]  Jian Sun,et al.  Nonparametric and Semi-Parametric Sensor Recovery in Multichannel Condition Monitoring Systems , 2011, IEEE Transactions on Automation Science and Engineering.

[28]  Emmanuel Ramasso,et al.  Investigating Computational Geometry for Failure Prognostics in Presence of Imprecise Health Indicator: Results and Comparisons on C-MAPSS Datasets , 2014 .

[29]  M. Bodruzzaman,et al.  Decision fusion software system for turbine engine fault diagnostics , 2012, 2012 Proceedings of IEEE Southeastcon.

[30]  Donghua Zhou,et al.  Online probabilistic operational safety assessment of multi-mode engineering systems using Bayesian methods , 2013, Reliab. Eng. Syst. Saf..

[31]  Noureddine Zerhouni,et al.  Strategies to Face Imbalanced and Unlabelled Data in Phm Applications , 2013 .

[32]  K. Goebel,et al.  Metrics for evaluating performance of prognostic techniques , 2008, 2008 International Conference on Prognostics and Health Management.

[33]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[34]  Michael Pecht,et al.  Study of ensemble learning-based fusion prognostics , 2010, 2010 Prognostics and System Health Management Conference.

[35]  Jianbo Yu,et al.  A nonlinear probabilistic method and contribution analysis for machine condition monitoring , 2013 .

[36]  G. V. Puskorius,et al.  A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification , 1998, Proc. IEEE.

[37]  Khanh Le Son,et al.  Remaining useful life estimation on the non-homogenous gamma with noise deterioration based on Gibbs filtering: A case study , 2012, 2012 IEEE Conference on Prognostics and Health Management.

[38]  Hanz Richter.  Engine Models and Simulation Tools , 2012 .

[39]  Noureddine Zerhouni,et al.  Features Selection Procedure for Prognostics: An Approach Based on Predictability , 2012 .

[40]  Jianbo Yu,et al.  A similarity-based prognostics approach for Remaining Useful Life estimation of engineered systems , 2008, 2008 International Conference on Prognostics and Health Management.

[41]  Chao Hu,et al.  A Copula-based sampling method for data-driven prognostics and health management , 2013, 2013 IEEE Conference on Prognostics and Health Management (PHM).

[42]  Benoît Iung,et al.  Remaining useful life estimation based on stochastic deterioration models: A comparative study , 2013, Reliab. Eng. Syst. Saf..

[43]  G. Klir,et al.  Uncertainty-based information: Elements of generalized information theory (studies in fuzziness and soft computing). , 1998 .

[44]  Byeng D. Youn,et al.  A generic probabilistic framework for structural health prognostics and uncertainty management , 2012 .

[45]  Cairo Lucio Nascimento Junior,et al.  GFRBS-PHM: A Genetic Fuzzy Rule-Based System for PHM with improved interpretability , 2013, 2013 IEEE Conference on Prognostics and Health Management (PHM).

[46]  Noureddine Zerhouni,et al.  Joint Prediction of Continuous and Discrete States in Time-Series Based on Belief Functions , 2013, IEEE Transactions on Cybernetics.

[47]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[48]  Saleh Zein-Sabatto,et al.  Statistical approach to online prognostics of turbine engine components , 2013, 2013 Proceedings of IEEE Southeastcon.

[49]  Hugo Valadares Siqueira,et al.  Echo State Networks and Extreme Learning Machines: A Comparative Study on Seasonal Streamflow Series Prediction , 2012, ICONIP.

[50]  Sankalita Saha,et al.  Metrics for Offline Evaluation of Prognostic Performance , 2021, International Journal of Prognostics and Health Management.

[51]  Asok Ray,et al.  Data-Driven Fault Detection in Aircraft Engines With Noisy Sensor Measurements , 2011 .

[52]  Peng Xiyuan,et al.  Sensor Selection with Grey Correlation Analysis for Remaining Useful Life Evaluation , 2012 .

[53]  Thierry Denoeux,et al.  Resample and combine: an approach to improving uncertainty representation in evidential pattern classification , 2003, Inf. Fusion.

[54]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[55]  Pingfeng Wang,et al.  Failure diagnosis using deep belief learning based health state classification , 2013, Reliab. Eng. Syst. Saf..

[56]  David Siegel.  Evaluation of health assessment techniques for rotating machinery , 2009 .

[57]  Plamen Angelov,et al.  Evolving Intelligent Systems: Methodology and Applications , 2010 .

[58]  Noureddine Zerhouni,et al.  Connexionist-Systems-Based Long Term Prediction Approaches for Prognostics , 2012, IEEE Transactions on Reliability.