Some soft measures for performance analysis: The “core” dimensions of expert system quality

Abstract The post-implementation evaluation of new technologies such as artificial intelligence, multimedia or parallel computing systems which are in operation is a much neglected problem. There is a lack of methods, techniques and tools for the practitioner to analyze the performance such systems. In earlier work, we have presented a framework and methodology that described a socio-technical approach for evaluating expert systems. In short, we determined the features and characteristics of expert systems that are most critical for their “implementation success”. This was the result of empirical evidence from a first-ever field study of expert systems in production. In this paper, specifically tailored to an audience of generalists, we retrospectively put our work into the framework of performance measurement and analysis. The end-result of our investigations is that we have definitional clarity as to how we may alternately evaluate such decision support systems.

[1]  Ravi S. Sharma,et al.  Evaluating expert systems: the socio‐technical dimensions of quality , 1992 .

[2]  Ravi S. Sharma,et al.  Towards a diagnostic instrument for assessing the quality of expert systems , 1990, SIGBDP '90.

[3]  Henry C. Lucas,et al.  Information Systems Concepts for Management , 1982 .

[4]  Blake Ives,et al.  The measurement of user information satisfaction , 1983, CACM.

[5]  Erik Hollnagel,et al.  Evaluation of Expert Systems , 1989 .

[6]  Charlie Ellis,et al.  Expert Knowledge and Explanation. The Knowledge--Language Interface , 1989 .

[7]  Jayant V. Saraph,et al.  An Instrument for Measuring the Critical Factors of Quality Management , 1989 .

[8]  Robert M. O’Keefe,et al.  The evaluation of decision-aiding systems: Guidelines and methods , 1989, Inf. Manag..

[9]  Sid L. Huff,et al.  Implementing Decision Support Systems: Correlates Of User Satisfaction And System Usage , 1990 .

[10]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[11]  E. Mumford,et al.  Managerial expert systems and organizational change: some critical research issues , 1987 .

[12]  Janet A. Sniezek,et al.  Defining Decision Support Systems: An Empirical Assessment Of End-User Satisfaction , 1989 .

[13]  Giovanni Guida,et al.  Topics in Expert System Design: Methodologies and Tools , 1989 .

[14]  Mark S. Fox,et al.  AI and expert system myths, legends, and facts , 1990, IEEE Expert.

[15]  J. Rothenberg.  Evaluating expert system tools: a framework and methodology , 1987 .

[16]  Edward A. Feigenbaum,et al.  The rise of the expert company , 1988 .

[17]  Ravi S. Sharma,et al.  A socio-technical model for deploying expert systems. I. The general theory , 1991 .

[18]  Frederick Hayes-Roth,et al.  Building expert systems , 1983, Advanced book program.

[19]  R. McLeod,et al.  Perceptions of system effectiveness as viewed by executives, users, and information specialists , 1988, [1988] Proceedings of the Twenty-First Annual Hawaii International Conference on System Sciences. Volume III: Decision Support and Knowledge Based Systems Track.

[20]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[21]  Dennis F. Galletta,et al.  Some Cautions on the Measurement of User Information Satisfaction , 1989 .

[22]  Tor Guimaraes,et al.  Measuring top management satisfaction with the MIS department , 1988 .

[23]  Wanda J. Orlikowski,et al.  A Short Form Measure of User Information Satisfaction: A Psychometric Evaluation and Notes on Use , 1987, J. Manag. Inf. Syst..

[24]  R. Hirschheim,et al.  Critical issues in information systems research , 1987 .

[25]  I. Bernstein.  Applied Multivariate Analysis , 1988 .

[26]  Ephraim R. McLean,et al.  Information Systems Success: The Quest for the Dependent Variable , 1992, Inf. Syst. Res..

[27]  Philip Klahr,et al.  Evaluation of expert systems: issues and case studies , 1983 .

[28]  Sammy W. Pearson,et al.  Development of a Tool for Measuring and Analyzing Computer User Satisfaction , 1983 .

[29]  Ramesh Sharda,et al.  Decision support system effectiveness: a review and an empirical test , 1988 .

[30]  E. Trist.  The Evolution of Socio-Technical Systems: A Conceptual Framework and an Action Research Program , 1981 .

[31]  William J. Doll,et al.  The Measurement of End-User Computing Satisfaction , 1988, MIS Q..

[32]  George F. Luger,et al.  Artificial Intelligence and the Design of Expert Systems , 1990 .