Metrics for Offline Evaluation of Prognostic Performance

Prognostic performance evaluation has gained significant attention in the past few years. * Currently, prognostics concepts lack standard definitions and suffer from ambiguous and inconsistent interpretations. This lack of standards is in part due to the varied enduser requirements for different applications, time scales, available information, domain dynamics, etc. to name a few. The research community has used a variety of metrics largely based on convenience and their respective requirements. Very little attention has been focused on establishing a standardized approach to compare different efforts. This paper presents several new evaluation metrics tailored for prognostics that were recently introduced and were shown to effectively evaluate various algorithms as compared to other conventional metrics. Specifically, this paper presents a detailed discussion on how these metrics should be interpreted and used. These metrics have the capability of incorporating probabilistic uncertainty estimates from prognostic algorithms. In addition to quantitative assessment they also offer a comprehensive visual perspective that can be used in designing the prognostic system. Several methods are suggested to customize these metrics for different applications. Guidelines are provided to help choose one method over another based on distribution characteristics. Various issues faced by prognostics and its performance evaluation are discussed followed by a formal notational framework to help standardize subsequent developments.

[1]  Abhinav Saxena,et al.  - 1-A COMPARISON OF THREE DATA-DRIVEN TECHNIQUES FOR PROGNOSTICS , 2008 .

[2]  F. Mosteller,et al.  Understanding robust and exploratory data analysis , 1985 .

[3]  Hugh McManus,et al.  A framework for understanding uncertainty and its mitigation and exploitation in complex systems , 2006, IEEE Engineering Management Review.

[4]  K. Goebel,et al.  Metrics for evaluating performance of prognostic techniques , 2008, 2008 International Conference on Prognostics and Health Management.

[5]  George J. Vachtsevanos,et al.  A particle-filtering approach for on-line fault diagnosis and failure prognosis , 2009 .

[6]  P. Sandborn,et al.  The analysis of Return on Investment for PHM applied to electronic systems , 2008, 2008 International Conference on Prognostics and Health Management.

[7]  Xuefei Guan,et al.  Entropy-based probabilistic fatigue damage prognosis and algorithmic performance comparison , 2009 .

[8]  G. Kacprzynski,et al.  Advances in uncertainty representation and management for particle filtering applied to prognostics , 2008, 2008 International Conference on Prognostics and Health Management.

[9]  K. Goebel,et al.  Standardizing research methods for prognostics , 2008, 2008 International Conference on Prognostics and Health Management.

[10]  Michael Sherman.  Probability and Statistics for Engineering and the Sciences (6th ed.) , 2006 .

[11]  Scott Poll,et al.  A Survey of Health Management User Objectives Related to Diagnostic and Prognostic Metrics , 2009 .

[12]  J. Banks,et al.  Cost Benefit Analysis for Asset Health Management Technology , 2007, 2007 Annual Reliability and Maintainability Symposium.

[13]  Liang Tang,et al.  A Novel RSPF Approach to Prediction of High-Risk, Low-Probability Failure Events , 2009 .

[14]  Sankaran Mahadevan,et al.  Uncertainty Quantification in Fatigue Damage Prognosis , 2009 .

[15]  Stephen J. Engel,et al.  Prognostics, the real issues involved with predicting life remaining , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[16]  George Vachtsevanos,et al.  Methodologies for uncertainty management in prognostics , 2009, 2009 IEEE Aerospace conference.

[17]  Chris Drummond,et al.  Reverse-Engineering Costs : How much will a Prognostic Algorithm save ? , 2008 .

[18]  Sankalita Saha,et al.  On Applying the Prognostic Performance Metrics , 2009 .

[19]  Sankalita Saha,et al.  Evaluating algorithm performance metrics tailored for prognostics , 2009, 2009 IEEE Aerospace conference.

[20]  Raphael T. Haftka,et al.  Reducing Uncertainty in Damage Growth Properties by Structural Health Monitoring , 2009 .

[21]  Kai Goebel,et al.  A Survey of Artificial Intelligence for Prognostics , 2007, AAAI Fall Symposium: Artificial Intelligence for Prognostics.

[22]  Chunsheng Yang,et al.  Model evaluation for prognostics: estimating cost saving for the end users , 2007, Sixth International Conference on Machine Learning and Applications (ICMLA 2007).

[23]  Jay L. Devore,et al.  Probability and statistics for engineering and the sciences , 1982 .

[24]  Kai Goebel,et al.  Modeling Li-ion Battery Capacity Depletion in a Particle Filtering Framework , 2009 .

[25]  Angel R. Martinez,et al.  : Exploratory data analysis with MATLAB ® , 2007 .

[26]  Keung-Chi Ng,et al.  Uncertainty management in expert systems , 1990, IEEE Expert.

[27]  Mark Schwabacher,et al.  A Survey of Data -Driven Prognostics , 2005 .

[28]  M. Carrasco,et al.  A study of the impact of prognostic errors on system performance , 2006, RAMS '06. Annual Reliability and Maintainability Symposium, 2006..

[29]  K. Pipe.  Practical prognostics for Condition Based Maintenance , 2008, 2008 International Conference on Prognostics and Health Management.

[30]  T. Yoneyama,et al.  Prognostics performance metrics and their relation to requirements, design, verification and cost-benefit , 2008, 2008 International Conference on Prognostics and Health Management.

[31]  J.W. Hines,et al.  Prognostic algorithm categorization with PHM Challenge application , 2008, 2008 International Conference on Prognostics and Health Management.

[32]  K. Goebel,et al.  Prognostic information fusion for constant load systems , 2005, 2005 7th International Conference on Information Fusion.