A multifaceted approach to evaluating expert systems

A multifaceted approach to evaluating expert systems is overviewed. This approach has three facets: a technical facet, for “looking inside the black box”; an empirical facet, for assessing the system’s impact on performance; and a subjective facet, for obtaining users’ judgments about the system. Such an approach is required to test the system against the different types of criteria of interest to sponsors and users and is consistent with evolving lifecycle paradigms. Moreover, such an approach leads to the application of different evaluation methods to answer different types of evaluation questions. Different evaluation methods for each facet are overviewed.

[1]  George P. Huber,et al.  DECISION SUPPORT SYSTEMS FOR ILL‐STRUCTURED PROBLEMS: AN EMPIRICAL STUDY , 1987 .

[2]  D. E. Bell,et al.  Conflicting Objectives in Decisions , 1978 .

[3]  Andrew P. Sage,et al.  Concise encyclopedia of information processing in systems &organizations , 1990 .

[4]  Alun Preece,et al.  Towards a methodology for evaluating expert systems , 1990 .

[5]  Elizabeth Cholawsky.  Beating the prototyping blues , 1988 .

[6]  Keith Levi,et al.  Expert systems should be more accurate than human experts: evaluation procedures from human judgement and decision making , 1989, IEEE Trans. Syst. Man Cybern..

[7]  Larry Kerschberg,et al.  A system development methodology for knowledge-based systems , 1989, IEEE Trans. Syst. Man Cybern..

[8]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[9]  Terry Winograd,et al.  Expert Systems: How Far Can They Go? , 1985, IJCAI.

[10]  Barry W. Boehm,et al.  Verifying and Validating Software Requirements and Design Specifications , 1989, IEEE Software.

[11]  T. Saaty,et al.  The Analytic Hierarchy Process , 1985 .

[12]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[13]  R. Yin.  Case Study Research: Design and Methods , 1984 .

[14]  John F. Passafiume,et al.  Software testing and evaluation , 1987 .

[15]  Leonard Adelman,et al.  Evaluating decision support and expert systems , 1991 .

[16]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[17]  Dianne C. Berry,et al.  Evaluating expert systems , 1990 .

[18]  Osman Balci,et al.  Validating Expert System Performance , 1987, IEEE Expert.

[19]  G. Huber.  Managerial decision making , 1980 .

[20]  J. Casey,et al.  Picking the right expert system application , 1989 .

[21]  Frederick W. Rook,et al.  The knowledge acquisition activity matrix: a systems engineering conceptual framework , 1989, IEEE Trans. Syst. Man Cybern..

[22]  Stephen J. Andriole.  Microcomputer decision support systems : design, implementation, and evaluation , 1986 .

[23]  William G. Cochran,et al.  Experimental Designs, 2nd Edition , 1950 .

[24]  Uma G. Gupta.  Validating and Verifying Knowledge-Based Systems , 1991 .

[25]  Paul E. Lehner,et al.  User and R&D specialist evaluation of decision-support systems , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[26]  Robert W. Bailey,et al.  Human performance engineering: using human factors/ergonomics to achieve computer system usability (2nd ed.) , 1989 .

[27]  Jacob W. Ulvila,et al.  Testing knowledge-based systems: The state of the practice and suggestions for improvement , 1990 .

[28]  Ronald J. Ebert,et al.  Bootstrapping the security analyst. , 1978 .

[29]  Allen S. Lee.  A Scientific Methodology for MIS Case Studies , 1989, MIS Q..

[30]  Lawrence M. Fagan,et al.  Antimicrobial selection by a computer. A blinded evaluation by infectious diseases experts. , 1979, JAMA.

[31]  Efraim Turban,et al.  Decision Support and Expert Systems: Management Support Systems , 1990 .

[32]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[33]  Yue Kang,et al.  A tool for detecting expert-system errors , 1989 .

[34]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[35]  Jay Liebowitz,et al.  Useful approach for evaluating expert systems , 1986 .

[36]  Leonard Adelman,et al.  Experiments, quasi-experiments, and case studies: A review of empirical methods for evaluating decision support systems , 1991, IEEE Trans. Syst. Man Cybern..

[37]  Lewis R. Goldberg,et al.  Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. , 1970 .

[38]  R. Hogarth,et al.  Unit weighting schemes for decision making , 1975 .

[39]  Paul E. Lehner.  Toward an empirical approach to evaluating the knowledge base of an expert system , 1989, IEEE Trans. Syst. Man Cybern..

[40]  Frederick Hayes-Roth,et al.  Building expert systems , 1983, Advanced book program.

[41]  Gary Berg-Cross,et al.  Performance Evaluation of Artificial Intelligence Systems , 1987 .

[42]  Robert M. O'Keefe,et al.  An integrative model of expert system verification and validation , 1990 .

[43]  Barry Kirwan,et al.  A Guide To Task Analysis: The Task Analysis Working Group , 1992 .

[44]  James R. Slagle,et al.  A Method for Evaluating Candidate Expert System Applications , 1988, AI Mag..

[45]  H. Raiffa,et al.  Decisions with Multiple Objectives , 1993 .

[46]  Bruce Marcot,et al.  Testing your knowledge base , 1987 .

[47]  Richard E. Fairley,et al.  Software engineering concepts , 1985, McGraw-Hill series in software engineering and technology.

[48]  P. Hoffman.  The paramorphic representation of clinical judgment. , 1960, Psychological bulletin.

[49]  Gordon F. Pitz,et al.  Utilization-Oriented Evaluation of Decision Support Systems , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[50]  Derek L. Nazareth,et al.  Issues in the Verification of Knowledge in Rule-Based Systems , 1989, Int. J. Man Mach. Stud..

[51]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[52]  Paul Harmon,et al.  Expert systems: tools and applications , 1988 .

[53]  Tin A. Nguyen,et al.  Knowledge base verification , 1987 .

[54]  Ching Y. Suen,et al.  Expert system evaluation techniques: a selected bibliography , 1991 .

[55]  John L. Bennett,et al.  Building Decision Support Systems , 1983 .