Validation techniques for case-based reasoning systems

By their nature, case-based reasoning (CBR) systems, have a built-in set of test cases in their case library. Effective use of this feature can facilitate the validation process by minimizing the involvement of domain experts. This can reduce the cost of the validation procedure, and eliminate the subjectivity introduced by experts. This paper proposes a validation technique that makes use of the system's own case library to validate a CBR system. Called the case library subset test (CLST) technique, it evaluates the correctness of the retrieval and adaptation functions of the CBR engine with respect to the domain as represented by the case library. It is composed of three phases, 1) the retrieval test, 2) the adaptation test, and 3) the domain coverage test. A complete description of the technique and an application of it to validate an existing CBR system are included in this paper.

[1]  M O'Neil,et al.  Evaluating and validating very large knowledge-based systems. , 1990, Medical informatics = Medecine et informatique.

[2]  Daniel E. O'Leary,et al.  VALIDATION OF EXPERT SYSTEMS- WITH APPLICATIONS TO AUDITING AND ACCOUNTING EXPERT SYSTEMS* , 1987 .

[3]  Avelino J. Gonzalez,et al.  Performance evaluation of a large diagnostic expert system using a heuristic test case generator , 1996 .

[4]  Jaime G. Carbonell,et al.  Derivational analogy: a theory of reconstructive problem solving and expertise acquisition , 1993 .

[5]  Ed P. Andert,et al.  Integrated Knowledge-Based System Design and Validation for Solving Problems in Uncertain Environments , 1992, Int. J. Man Mach. Stud..

[6]  Neli P. Zlatareva,et al.  A framework for verification, validation, and refinement of knowledge bases: The VVR system , 1994, Int. J. Intell. Syst..

[7]  R. A. Stachowitz,et al.  A report on the Expert Systems Validation Associate (EVA) , 1990 .

[8]  Kristian J. Hammond,et al.  Case-Based Planning: Viewing Planning as a Memory Task , 1989 .

[9]  Ashwin Ram,et al.  A Theory of Questions and Question Asking , 1991 .

[10]  Ching Y. Suen,et al.  Verifying, validating, and measuring the performance of expert systems , 1990 .

[11]  Evangelos Simoudis,et al.  Validated Retrieval in Case-Based Reasoning , 1990, AAAI.

[12]  Uma G. Gupta.  Validating and Verifying Knowledge-Based Systems , 1991 .

[13]  Janet L. Kolodner,et al.  Case-Based Reasoning , 1989, IJCAI 1989.

[14]  Ray Bareiss,et al.  Concept Learning and Heuristic Classification in WeakTtheory Domains , 1990, Artif. Intell..

[15]  Christopher J. R. Green,et al.  Verification and validation of expert systems , 1991 .

[16]  Kevin D. Ashley,et al.  Compare and Contrast: A Test of Expertise , 1987, AAAI.

[17]  Richard G. Hamlet,et al.  Implementing prototype testing tools , 1995, Softw. Pract. Exp..

[18]  Daniel E. O'Leary,et al.  Verification and validation of case-based systems , 1993 .

[19]  A. Gonzalez,et al.  A case-based reasoning approach to real estate property appraisal , 1992 .

[20]  Ortiz Raymond Laureano.  Application of case-based reasoning techniques to the automation of single-family residential property appraisals , 1990 .

[21]  Evangelos Simoudis,et al.  Using case-based retrieval for customer technical support , 1992, IEEE Expert.

[22]  Osman Balci,et al.  Validating Expert System Performance , 1987, IEEE Expert.