Evaluating Performance and Quality of Knowledge-Based Systems: Foundation and Methodology

A survey of knowledge-based system (KBS) evaluation methods is presented. The authors argue that these methods are partial, poorly systematic, and not easily applicable. An approach to KBS evaluation that comprises a precise definition of the concepts of performance and quality, a general evaluation methodology, and a set of criteria to support its practical application is presented. The proposed approach has been tried only partially and with rather simple test cases. >

[1]  James A. Reggia,et al.  Evaluation of Medical Expert Systems: A Case Study in Performance Assessment , 1988 .

[2]  Philip Klahr,et al.  Evaluation of expert systems: issues and case studies , 1983 .

[3]  Erik Hollnagel,et al.  Evaluation of Expert Systems , 1989 .

[4]  Mary Ann Goodwin,et al.  A software engineering approach to expert system design and verification , 1988 .

[5]  Ronald J. Brachman,et al.  "I Lied About the Trees", Or, Defaults and Definitions in Knowledge Representation , 1985, AI Mag..

[6]  Edward H. Shortliffe,et al.  The problem of evaluation , 1984 .

[7]  Randall Davis,et al.  Use of meta level knowledge in the construction and maintenance of large knowledge bases , 1976 .

[8]  Paul Harmon,et al.  Creating Expert Systems for Business and Industry , 1990 .

[9]  Alun Preece,et al.  Towards a methodology for evaluating expert systems , 1990 .

[10]  Daniel E. O'Leary,et al.  VALIDATION OF EXPERT SYSTEMS- WITH APPLICATIONS TO AUDITING AND ACCOUNTING EXPERT SYSTEMS* , 1987 .

[11]  John Gaschnig,et al.  Preliminary Performance Analysis of the PROSPECTOR Consultant System for Mineral Exploration , 1979, IJCAI.

[12]  Keith Levi,et al.  Expert systems should be more accurate than human experts: evaluation procedures from human judgement and decision making , 1989, IEEE Trans. Syst. Man Cybern..

[13]  Walton A. Perkins,et al.  Checking an Expert Systems Knowledge Base for Consistency and Completeness , 1985, IJCAI.

[14]  Donald W. Loveland,et al.  Detecting Ambiguity: An Example in Knowledge Evaluation , 1983, IJCAI.

[15]  Michael Goul,et al.  Validating expert systems , 1990, IEEE Expert.

[16]  Edward H. Shortliffe,et al.  An approach to verifying completeness and consistency in a rule-expert system , 1989 .

[17]  Rolf Stachowitz,et al.  Building validation tools for knowledge-based systems , 1987 .

[18]  Bruce Marcot,et al.  Testing your knowledge base , 1987 .

[19]  Mark Chignell,et al.  Expert Systems For Experts , 1988 .

[20]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[21]  Lawrence Birnbaum,et al.  The architecture of expert systems , 1983 .

[22]  Jeremy Wyatt,et al.  The evaluation of clinical decision support systems: a discussion of the methodology used in the ACORN project , 1987 .

[23]  Giovanni Guida,et al.  Building Expert Systems: From Life Cycle to Development Methodology , 1989 .

[24]  Osman Balci,et al.  Validating Expert System Performance , 1987, IEEE Expert.

[25]  Alan Bundy,et al.  How to improve the reliability of expert systems , 1988 .

[26]  Edward H. Shortliffe,et al.  Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (The Addison-Wesley series in artificial intelligence) , 1984 .

[27]  Gevarter,et al.  Overview of Expert Systems , 1982 .

[28]  Edsger W. Dijkstra,et al.  A Discipline of Programming , 1976 .

[29]  Donald A. Waterman,et al.  A Guide to Expert Systems , 1986 .

[30]  Tomek Strzalkowski,et al.  Designing and automating the quality assessment of a knowledge-based. system: The initial Automated academic advisor experience , 1984 .

[31]  Victor W. Marek,et al.  Completeness and Consistency in Knowledge Base Systems , 1986, Expert Database Conf..

[32]  Tin A. Nguyen,et al.  Knowledge base verification , 1987 .

[33]  Jay Liebowitz,et al.  Useful approach for evaluating expert systems , 1986 .

[34]  Balakrishnan Chandrasekaran,et al.  On evaluating ai systems for medical diagnosis , 1989 .

[35]  Harold J. Steudel,et al.  A Decision-Table-Based Processor for Checking Completeness and Consistency in Rule-Based Expert Systems , 1987, Int. J. Man Mach. Stud..

[36]  Frederick Hayes-Roth,et al.  Building expert systems , 1983, Advanced book program.

[37]  Frederick Hayes-Roth.  Towards Benchmarks for Knowledge Systems and Their Implications for Data Engineering , 1989, IEEE Trans. Knowl. Data Eng..

[38]  Michael S. H. Heng,et al.  Why evolutionary development of expert systems appears to work , 1987, Future Gener. Comput. Syst..

[39]  Allen Ginsberg.  Knowledge-Base Reduction: A New Approach to Checking knowledge Bases for Inconsistency and Redundancy , 1988, AAAI.

[40]  Sheila A. Ryan,et al.  Validating the Knowledge in a Computer-Based Consultant for Nursing Care , 1987 .

[41]  Edward H. Shortliffe,et al.  An Approach to Verifying Completeness and Consistency in a Rule-Based Expert System , 1982, AI Mag..

[42]  Mike Greenwell.  Knowledge engineering for expert systems , 1988 .

[43]  Rolf Stachowitz,et al.  Validation of knowledge-based systems , 1987 .

[44]  Rolf Stachowitz,et al.  Validation of expert systems , 1988 .

[45]  Ching Y. Suen,et al.  Verifying, validating, and measuring the performance of expert systems , 1990 .

[46]  Christopher J. R. Green,et al.  Verification and validation of expert systems , 1991 .

[47]  Allen Newell,et al.  The Knowledge Level , 1989, Artif. Intell..

[48]  Dianne C. Berry,et al.  Evaluating expert systems , 1990 .

[49]  B. Chandrasekaran,et al.  Generic Tasks in Knowledge-Based Reasoning: High-Level Building Blocks for Expert System Design , 1986, IEEE Expert.

[50]  John Rushby,et al.  Quality Measures and Assurance for AI (Artificial Intelligence) Software , 1988 .

[51]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[52]  John C. Cherniavsky,et al.  Validation, Verification, and Testing of Computer Software , 1982, CSUR.

[53]  G. Guida,et al.  Evaluation of natural language processing systems: Issues and approaches , 1986, Proceedings of the IEEE.