A framework for systematic benchmarking of monitoring and diagnostic systems

In this paper, we present an architecture and a formal framework to be used for systematic benchmarking of monitoring and diagnostic systems and for producing comparable performance assessments of different diagnostic technologies. The framework defines a number of standardized specifications, which include a fault catalog, a library of modular test scenarios, and a common protocol for gathering and processing diagnostic data. At the center of the framework are 13 benchmarking metric definitions. The calculation of metrics is illustrated on a probabilistic model-based diagnosis algorithm utilizing Bayesian reasoning techniques. The diagnosed system is a real-world electrical power system, namely the Advanced Diagnostics and Prognostics Testbed (ADAPT) developed and located at the NASA Ames Research Center. The proposed benchmarking approach shows how to generate realistic diagnostic data sets for large-scale, complex engineering systems, and how the generated experimental data can be used to enable ldquoapples to applesrdquo assessments of the effectiveness of different diagnostic and monitoring algorithms.

[1]  Ole J. Mengshoel,et al.  Sensor Validation using Bayesian Networks , 2008 .

[2]  Gautam Biswas,et al.  Bayesian Fault Detection and Diagnosis in Dynamic Systems , 2000, AAAI/IAAI.

[3]  Adnan Darwiche,et al.  Compiling Bayesian Networks Using Variable Elimination , 2007, IJCAI.

[4]  M.J. Roemer,et al.  Validation and verification of prognostic and health management technologies , 2005, 2005 IEEE Aerospace Conference.

[5]  Krishna R. Pattipati,et al.  A Unified Metric for Fault Detection and Isolation in Engines , 2006 .

[6]  Adnan Darwiche,et al.  Compiling Bayesian Networks with Local Structure , 2005, IJCAI.

[7]  David L. Iverson.  Inductive System Health Monitoring , 2004, IC-AI.

[8]  X. Koutsoukos,et al.  Evaluation, Selection, and Application of Model-Based Diagnosis Tools and Approaches , 2007 .

[9]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[10]  Ole J. Mengshoel,et al.  Diagnosing Faults in Electrical Power Systems of Spacecraft and Aircraft , 2008, AAAI.

[11]  J. Kurien,et al.  Costs and Benefits of Model-based Diagnosis , 2008, 2008 IEEE Aerospace Conference.

[12]  Ole J. Mengshoel,et al.  Advanced Diagnostics and Prognostics Testbed , 2007 .

[13]  Jon Hull,et al.  Metrics for Evaluating the Accuracy of Diagnostic Fault Detection Systems , 2004 .

[14]  A. Darwiche,et al.  Complexity Results and Approximation Strategies for MAP Explanations , 2011, J. Artif. Intell. Res..

[15]  William A. Maul,et al.  Sensor Data Qualification for Autonomous Operation of Space Systems , 2006, AAAI Fall Symposium: Spacecraft Autonomy.

[16]  Adnan Darwiche,et al.  A differential approach to inference in Bayesian networks , 2000, JACM.

[17]  Fernando Figueroa,et al.  Rocket Testing and Integrated System Health Management , 2006 .

[18]  Ole J. Mengshoel.  Macroscopic Models of Clique Tree Growth for Bayesian Networks , 2007, AAAI.

[19]  David C. Wilkins,et al.  Controlled generation of hard and easy Bayesian networks: Impact on maximal clique size in tree clustering , 2006, Artif. Intell..

[20]  Xenofon D. Koutsoukos,et al.  A Comprehensive Diagnosis Methodology for Complex Hybrid Systems: A Case Study on Spacecraft Power Distribution Systems , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[21]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[22]  D.L. Iverson.  System Health Monitoring for Space Mission Operations , 2008, 2008 IEEE Aerospace Conference.

[23]  Ole J. Mengshoel,et al.  Designing Resource-Bounded Reasoners using Bayesian Networks: System Health Monitoring and Diagnosis , 2007 .

[24]  Z. Williams.  Benefits of IVHM: an analytical approach , 2006, 2006 IEEE Aerospace Conference.

[25]  Timothy W. Bickmore.  A probabilistic approach to sensor data validation , 1992 .

[26]  Lihui Wang,et al.  Condition monitoring and control for intelligent manufacturing , 2006 .

[27]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[28]  Joseba Quevedo,et al.  Introduction to the DAMADICS actuator FDI benchmark study , 2006 .

[29]  I. Turner,et al.  On Quantifying Cost-Benefit of ISHM in Aerospace Systems , 2007, 2007 IEEE Aerospace Conference.

[30]  Sriram Narasimhan,et al.  Combining Particle Filters and Consistency-Based Approaches for Monitoring and Diagnosis of Stochastic Hybrid Systems , 2004 .

[31]  Ole J. Mengshoel,et al.  Probabilistic Model-Based Diagnosis: An Electrical Power System Case Study , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[32]  M. S. Lebold,et al.  Development of performance and effectiveness metrics for gas turbine diagnostic technologies , 2002, Proceedings, IEEE Aerospace Conference.

[33]  R. L. Bickford,et al.  REAL-TIME SENSOR VALIDATION FOR AUTONOMOUS FLIGHT CONTROL , 1997 .

[34]  W E Vesely,et al.  Fault Tree Handbook , 1987 .

[35]  Du Zhang,et al.  Diagnosis of component failures in the Space Shuttle main engines using Bayesian belief networks: a feasibility study , 2002, 14th IEEE International Conference on Tools with Artificial Intelligence, 2002. (ICTAI 2002). Proceedings..