A quantitative analysis of the robustness of knowledge-based systems through degradation studies

The overall aim of this paper is to provide a general setting for quantitative quality measures of knowledge-based system behaviour that is widely applicable to many knowledge-based systems. We propose a general approach that we call degradation studies: an analysis of how system output changes as a function of degrading system input, such as incomplete or incorrect data or knowledge. To show the feasibility of our approach, we have applied it in a case study. We have taken a large and realistic vegetation-classification system, and have analysed its behaviour under various varieties of incomplete and incorrect input. This case study shows that degradation studies can reveal interesting and surprising properties of the system under study.

[1]  Frank Puppe,et al.  Wiederverwendbare Bausteine für eine konfigurierbare Diagnostik-Shell , 1994, Künstliche Intell..

[2]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[3]  Nigel Shadbolt,et al.  The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions , 1999, Int. J. Hum. Comput. Stud..

[4]  Alun D. Preece,et al.  Evaluation of verification tools for knowledge-based systems , 1997, Int. J. Hum. Comput. Stud..

[5]  Frank van Harmelen,et al.  Computing Approximate Diagnoses By Using Approximate Entailment , 1996, KR.

[6]  Mark S. Boddy,et al.  An Analysis of Time-Dependent Planning , 1988, AAAI.

[7]  Paul R. Cohen,et al.  Empirical methods for artificial intelligence , 1995, IEEE Expert.

[8]  Frank van Harmelen,et al.  Characterising approximate problem solving: by partially fulfilled pre- and postconditions , 1998, EUROVAV.

[9]  Frank Puppe,et al.  Wissensbasierte Diagnose- und Informationssysteme - Mit Anwendungen des Expertensystem-Shell-Baukastens D3 , 1996, Wissensbasierte Diagnose- und Informationssysteme.

[10]  Frank van Harmelen,et al.  Editorial: Evaluating knowledge engineering techniques , 1999, Int. J. Hum. Comput. Stud..

[11]  Frederick Hayes-Roth,et al.  Knowledge-based expert systems — The State of the Art in the US , 1984, The Knowledge Engineering Review.

[12]  Frank van Harmelen,et al.  Exploiting Domain Knowledge for Approximate Diagnosis , 1997, IJCAI.

[13]  Johann Schumann,et al.  NORA/HAMMR: making deduction-based software component retrieval practical , 1997, Proceedings 12th IEEE International Conference Automated Software Engineering.

[14]  Shlomo Zilberstein,et al.  Using Anytime Algorithms in Intelligent Systems , 1996, AI Mag..