Empirical Evaluation of Diagnostic Algorithm Performance Using a Generic Framework

A variety of rule-based, model-based and datadriven techniques have been proposed for detection and isolation of faults in physical systems. However, there have been few efforts to comparatively analyze the performance of these approaches on the same system under identical conditions. One reason for this was the lack of a standard framework to perform this comparison. In this paper we introduce a framework, called DXF, that provides a common language to represent the system description, sensor data and the fault diagnosis results; a run-time architecture to execute the diagnosis algorithms under identical conditions and collect the diagnosis results; and an evaluation component that can compute performance metrics from the diagnosis results to compare the algorithms. We have used DXF to perform an empirical evaluation of 13 diagnostic algorithms on a hardware testbed (ADAPT) at NASA Ames Research Center and on a set of synthetic circuits typically used as benchmarks in the model-based diagnosis community. Based on these empirical data we analyze the performance of each algorithm and suggest directions for future development.

[1]  Gautam Biswas,et al.  Bayesian Fault Detection and Diagnosis in Dynamic Systems , 2000, AAAI/IAAI.

[2]  M. S. Lebold,et al.  Development of performance and effectiveness metrics for gas turbine diagnostic technologies , 2002, Proceedings, IEEE Aerospace Conference.

[3]  Gautam Biswas,et al.  Designing Distributed Diagnosers for Complex Continuous Systems , 2009, IEEE Transactions on Automation Science and Engineering.

[4]  Donald W. Loveland,et al.  A machine program for theorem-proving , 2011, CACM.

[5]  Stephen P. Boyd,et al.  Relaxed maximum a posteriori fault identification , 2009, Signal Process..

[6]  M.J. Roemer,et al.  Validation and verification of prognostic and health management technologies , 2005, 2005 IEEE Aerospace Conference.

[7]  Gregory M. Provan,et al.  Computing Observation Vectors for Max-Fault Min-Cardinality Diagnoses , 2008, AAAI.

[8]  Alban Grastien,et al.  Wizards of Oz; Description of the 2009 DXC Entry , 2009 .

[9]  David L. Iverson.  Inductive System Health Monitoring , 2004, IC-AI.

[10]  Kenneth D. Forbus,et al.  Building Problem Solvers , 1993 .

[11]  Jon Hull,et al.  Metrics for Evaluating the Accuracy of Diagnostic Fault Detection Systems , 2004 .

[12]  J. Kurien,et al.  Costs and Benefits of Model-based Diagnosis , 2008, 2008 IEEE Aerospace Conference.

[13]  N. D. Freitas.  Rao-Blackwellised particle filtering for fault diagnosis , 2002 .

[14]  Joseba Quevedo,et al.  Introduction to the DAMADICS actuator FDI benchmark study , 2006 .

[15]  Burkhard Münker,et al.  Model-Based Failure Analysis with RODON , 2006, ECAI.

[16]  Jan Eric Larsson,et al.  Diagnosis Based on Explicit Means-End Models , 1996, Artif. Intell..

[17]  Z. Williams.  Benefits of IVHM: an analytical approach , 2006, 2006 IEEE Aerospace Conference.

[18]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[19]  Jinbo Huang,et al.  Hierarchical Diagnosis of Multiple Faults , 2007, IJCAI.

[20]  Michèle Basseville,et al.  Detection of abrupt changes: theory and application , 1993 .

[21]  G. Provan,et al.  The Lydia Approach to Combinational Model-Based Diagnosis , 2009 .

[22]  Eugene F. Schuster,et al.  On the negative hypergeometric distribution , 1987 .

[23]  Raymond Reiter,et al.  Characterizing Diagnoses and Systems , 1992, Artif. Intell..

[24]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[25]  Alexander Feldman,et al.  First International Diagnosis Competition - DXC'09 , 2009 .

[26]  F. Brglez,et al.  A neutral netlist of 10 combinational benchmark circuits and a target translator in FORTRAN , 1985 .

[27]  S. Poll,et al.  A framework for systematic benchmarking of monitoring and diagnostic systems , 2008, 2008 International Conference on Prognostics and Health Management.

[28]  Brian C. Williams,et al.  Diagnosing Multiple Faults , 1987, Artif. Intell..

[29]  Raymond Reiter,et al.  A Theory of Diagnosis from First Principles , 1986, Artif. Intell..

[30]  Donald L. Simon,et al.  Benchmarking Gas Path Diagnostic Methods: A Public Approach , 2008 .

[31]  Johan de Kleer.  Mininimum cardinality candidate generation , 2009 .

[32]  S. Narasimhan,et al.  HyDE – A General Framework for Stochastic and Hybrid Model-based Diagnosis , 2007 .

[33]  Gregory M. Provan,et al.  Computing Minimal Diagnoses by Greedy Stochastic Search , 2008, AAAI.

[34]  Heikki N. Koivo,et al.  Application of artificial neural networks in process fault diagnosis , 1991, Autom..

[35]  Krishna R. Pattipati,et al.  A Unified Metric for Fault Detection and Isolation in Engines , 2006 .

[36]  Wei Chen,et al.  Cost-Benefit Quantification of ISHM in Aerospace Systems , 2007 .

[37]  Ole J. Mengshoel,et al.  Designing Resource-Bounded Reasoners using Bayesian Networks: System Health Monitoring and Diagnosis , 2007 .

[38]  Mogens Blanke,et al.  A Ship Propulsion System as a Benchmark for Fault-tolerant Control , 1997 .

[39]  Janos Gertler,et al.  Fault detection and diagnosis in engineering systems , 1998 .

[40]  Ole J. Mengshoel,et al.  Advanced Diagnostics and Prognostics Testbed , 2007 .