First International Diagnosis Competition - DXC'09

A framework to compare and evaluate diagnosis algorithms (DAs) has been created jointly by NASA Ames Research Center and PARC. In this paper, we present the first concrete implementation of this framework as a competition called DXC 09. The goal of this competition was to evaluate and compare DAs in a common platform and to determine a winner based on diagnosis results. 12 DAs (model-based and otherwise) competed in this first year of the competition in 3 tracks that included industrial and synthetic systems. Specifically, the participants provided algorithms that communicated with the run-time architecture to receive scenario data and return diagnostic results. These algorithms were run on extended scenario data sets (different from sample set) to compute a set of pre-defined metrics. A ranking scheme based on weighted metrics was used to declare winners. This paper presents the systems used in DXC 09, description of faults and data sets, a listing of participating DAs, the metrics and results computed from running the DAs, and a superficial analysis of the results.

[1]  S. Narasimhan,et al.  HyDE – A General Framework for Stochastic and Hybrid Model-based Diagnosis , 2007 .

[2]  Gautam Biswas,et al.  Designing Distributed Diagnosers for Complex Continuous Systems , 2009, IEEE Transactions on Automation Science and Engineering.

[3]  J. E. Kristjansson,et al.  Hierarchical Diagnosis , 2006 .

[4]  Gregory M. Provan,et al.  Computing Observation Vectors for Max-Fault Min-Cardinality Diagnoses , 2008, AAAI.

[5]  Joseba Quevedo,et al.  Introduction to the DAMADICS actuator FDI benchmark study , 2006 .

[6]  Jurryt Pietersma,et al.  All Roads Lead to Fault Diagnosis: Model-Based Reasoning with LYDIA , 2006 .

[7]  M. S. Lebold,et al.  Development of performance and effectiveness metrics for gas turbine diagnostic technologies , 2002, Proceedings, IEEE Aerospace Conference.

[8]  Alexander Feldman,et al.  Towards a Framework for Evaluating and Comparing Diagnosis Algorithms , 2009 .

[9]  Ole J. Mengshoel,et al.  Advanced Diagnostics and Prognostics Testbed , 2007 .

[10]  Burkhard Münker,et al.  Model-Based Failure Analysis with RODON , 2006, ECAI.

[11]  F. Brglez,et al.  A neutral netlist of 10 combinational benchmark circuits and a target translator in FORTRAN , 1985 .

[12]  Stephen P. Boyd,et al.  Relaxed maximum a posteriori fault identification , 2009, Signal Process..

[13]  An Improved Approach for Generating Max-Fault Min-Cardinality Diagnoses , 2009 .

[14]  Ole J. Mengshoel,et al.  Designing Resource-Bounded Reasoners using Bayesian Networks: System Health Monitoring and Diagnosis , 2007 .

[15]  Donald L. Simon,et al.  Benchmarking Gas Path Diagnostic Methods: A Public Approach , 2008 .

[16]  Jinbo Huang,et al.  Hierarchical Diagnosis of Multiple Faults , 2007, IJCAI.

[17]  John P. Hayes,et al.  Unveiling the ISCAS-85 Benchmarks: A Case Study in Reverse Engineering , 1999, IEEE Des. Test Comput..