Evaluation of verification tools for knowledge-based systems

Validation has emerged as a significant problem in the development of knowledge based systems (KBS). Verification of KBS correctness and completeness has been cited as one of the most difficult aspects of validation. A number of software tools have been developed to perform such verification, but none of these are in widespread use. One of the reasons for this is that little quantitative evidence exists to demonstrate the effectiveness of the tools. This paper presents an experimental study of three KBS verification tools: a consistency checker, a completeness checker and a testing tool (for correctness). The tools are evaluated on their ability to reveal plausible faults seeded into a complex, realistic KBS application. The cost of using the tools is also measured. It is shown that each tool is independently effective at detecting certain kinds of fault and that the capabilities of the tools are complementary?a result not revealed by previous studies.

[1]  Alun D. Preece.  A New Approach to Detecting Missing Knowledge in Expert System Rule Bases , 1993, Int. J. Man Mach. Stud..

[2]  Jack R. Fraenkel.  Ask the Right Questions , 1966 .

[3]  Chris Culbert,et al.  State-of-the-practice in knowledge-based system verification and validation , 1991 .

[4]  Jean-Pierre H. Laurent.  Proposals for a Valid Terminology in KBS Validation , 1992, ECAI.

[5]  Allen Ginsberg.  Knowledge-Base Reduction: A New Approach to Checking knowledge Bases for Inconsistency and Redundancy , 1988, AAAI.

[6]  Daniel E. O'Leary.  Design, development and validation of expert systems: a survey of developers , 1991 .

[7]  Izak Benbasat,et al.  A framework for the validation of knowledge acquisition , 1989 .

[8]  Marc Ayel,et al.  A Conceptual Model for Building Sets of Test Samples for Knowledge Bases , 1990, ECAI.

[9]  Marc Ayel,et al.  SACCO-SYCOJET: two different ways of verifying knowledge-based systems , 1991 .

[10]  R. A. Stachowitz,et al.  A report on the Expert Systems Validation Associate (EVA) , 1990 .

[11]  Pedro Meseguer,et al.  Incremental Verification of Rule-Based Expert Systems , 1992, ECAI.

[12]  Marie-Christine Rousset,et al.  On the consistency of knowledge bases: the COVADIS system , 1988, Comput. Intell..

[13]  Johan de Kleer,et al.  An Assumption-Based TMS , 1987, Artif. Intell..

[14]  Amar Bendou.  CT-DATAGEN: A Constraint Based Test Data Generator , 1995, EUROVAV.

[15]  Lawrence M. Fagan,et al.  Validating the Knowledge Base of a Therapy Planning System , 1989, Methods of Information in Medicine.

[16]  Ruddy Lelouche,et al.  Test Case Generation using KBS Strategy , 1993, IJCAI.

[17]  John Rushby,et al.  Evaluation of an expert system for fault detection, isolation, and recovery in the manned maneuvering unit , 1990 .

[18]  Alun D. Preece,et al.  Verifying Knowledge Bases by Anomaly Detection: An Experience Report , 1992, ECAI.

[19]  Wei-Tek Tsai,et al.  Comparative evaluation of expert system testing methods , 1992, Proceedings Fourth International Conference on Tools with Artificial Intelligence TAI '92.

[20]  Marc Ayel,et al.  SYCOJET and SACCO, two tools for verifying expert systems , 1993 .