Use of the Critical Decision Method to Elicit Expert Knowledge: A Case Study in the Methodology of Cognitive Task Analysis

The Critical Decision Method (CDM) is an approach to cognitive task analysis. The method involves multiple-pass event retrospection guided by probe questions. The CDM has been used in the elicitation of expert knowledge in diverse domains and for applications including system development and instructional design. The CDM research illustrates the sorts of knowledge representation products that can arise from cognitive task analysis (e.g., Situation Assessment Records, time lines, decision requirements). The research also shows how one can approach methodological issues surrounding cognitive task analysis, including questions about data quality and method reliability, efficiency, and utility. As cognitive task analysis is used more widely in the elicitation, preservation, and dissemination of expert knowledge and is used more widely as the basis for the design of complex cognitive systems, and as projects move into even more field applications and real-world settings, the issues we discuss become increasingly critical.

[1]  George S. Howard,et al.  Why do people say nasty things about self-reports? , 1994 .

[2]  L. Shattuck,et al.  The Critical Incident Technique: 40 Years Later , 1994 .

[3]  Gary Klein,et al.  A literature review of analytical and naturalistic decision making , 1992 .

[4]  Nancy J. Cooke,et al.  Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing , 1996 .

[5]  Graeme Hirst,et al.  Knowledge and knowledge acquisition in the computational context , 1992 .

[6]  Jonathan S. Evans,et al.  Bias in human reasoning , 1990 .

[7]  Robert R. Hoffman,et al.  Constructivism versus realism, or constructivism and realism , 1994, J. Exp. Theor. Artif. Intell..

[8]  David S. Prerau.  Developing and managing expert systems - proven techniques for business and industry , 1989 .

[9]  B Crandall,et al.  Critical decision method: A technique for eliciting concrete assessment indicators from the intuition of NICU nurses , 1993, ANS. Advances in nursing science.

[10]  Paul J. Feltovich,et al.  Categorization and Representation of Physics Problems by Experts and Novices , 1981, Cogn. Sci..

[11]  Lisanne Bainbridge,et al.  Verbal reports as evidence of the process operator's knowledge , 1999, Int. J. Hum. Comput. Stud..

[12]  J. Shanteau.  Competence in experts: The role of task characteristics , 1992 .

[13]  G. Klein,et al.  Decision Making in Action: Models and Methods , 1993 .

[14]  Alex Goodall,et al.  The guide to expert systems , 1985 .

[15]  G. Klein,et al.  The Recognition-Primed Decision (RPD) Model: Looking Back, Looking Forward , 1997 .

[16]  Robert R. Hoffman,et al.  The Problem of Extracting the Knowledge of Experts from the Perspective of Experimental Psychology , 1987, AI Mag..

[17]  Gary Klein,et al.  Design Engineers and the Design Process: Decision Strategies and Human Factors Literature , 1986 .

[18]  Nancy J. Cooke,et al.  Eliciting Semantic Relations for Empirically Derived Networks , 1992, Int. J. Man Mach. Stud..

[19]  Andrew F. Monk,et al.  How and when to collect behavioural data , 1987 .

[20]  Mark Chignell,et al.  Strategic Issues in Knowledge Engineering , 1988 .

[21]  R. Glaser,et al.  Expertise in a complex skill: Diagnosing x-ray pictures. , 1988 .

[22]  Gary Klein,et al.  Designing for Performance: A Cognitive Systems Engineering Approach to Modifying an AWACS Human Computer Interface , 1993 .

[23]  Ranjit Bose,et al.  CMS: An intelligent knowledge-based tool for organizational procedure modeling and execution , 1995 .

[24]  George L. Kaempf,et al.  Decision Making in the AEGIS Combat Information Center , 1993 .

[25]  John Patrick,et al.  Training: Research and practice. , 1992 .

[26]  Marie E. Gomes,et al.  A Cognitive Systems Engineering Application for Interface Design , 1993 .

[27]  R. Fisher,et al.  Memory-Enhancing Techniques for Investigative Interviewing: The Cognitive Interview , 1992 .

[28]  Gary Klein,et al.  Representing cockpit crew decision making , 1991 .

[29]  Gary A. Klein,et al.  Applications of Analogical Reasoning , 1987 .

[30]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[31]  Ira Solomon,et al.  Effects of alternative sample space representations on the accuracy of auditors' uncertainty judgments , 1987 .

[32]  Walter G. Rolandi.  Knowledge engineering in practice , 1986 .

[33]  Beth Crandall,et al.  A comparative study of think-aloud and critical decision knowledge elicitation methods , 1989, SGAR.

[34]  Ian M. Neale,et al.  First generation expert systems: a review of knowledge acquisition methodologies , 1988, The Knowledge Engineering Review.

[35]  Jungduck Kim,et al.  A survey of knowledge acquisition techniques and their relevance to managerial problem domains , 1988, Decis. Support Syst..

[36]  Nancy J. Cooke,et al.  Measuring mental models: Choosing the right tools for the job , 1995 .

[37]  Julian S. Weitzenfeld,et al.  Knowledge Elicitation for Software Engineering Expertise , 1991, CSEE.

[38]  Nancy J. Cooke,et al.  Evaluating Mental Model Elicitation Methods , 1994 .

[39]  G A Klein,et al.  The use of analogues in comparability analysis. , 1982, Applied ergonomics.

[40]  Paul E. Spector.  Using self‐report questionnaires in OB research: A comment on the use of a controversial method , 1994 .

[41]  Larry E. Wood,et al.  Structuring interviews with experts during knowledge elicitation , 1993, Int. J. Intell. Syst..

[42]  John G. Gammack,et al.  Constructive interaction in knowledge engineering , 1990 .

[43]  Clifford R. Mynatt,et al.  On Scientific Thinking , 1981 .

[44]  Gary A. Klein,et al.  Using knowledge engineering to preserve corporate memory , 1992 .

[45]  Ray Paton,et al.  Towards an analysis and classification of approaches to knowledge acquisition from examination of textual metaphor , 1992 .

[46]  Alison L. Kidd,et al.  Knowledge Acquisition for Expert Systems , 1987 .

[47]  E. A. Fleishman,et al.  Toward a Taxonomy of Human Performance. , 1975 .

[48]  J. St B. T. Evans.  The knowledge elicitation problem: a psychological perspective , 1988 .

[49]  Jeffrey M. Bradshaw,et al.  Knowledge acquisition as modeling , 1993 .

[50]  Barbara Brown.  The taming of an expert: an anecdotal report , 1989, SGAR.

[51]  John Benjafield,et al.  Evidence that ‘thinking aloud’ constitutes an externalization of inner speech , 1969 .

[52]  Samuel W. Fernberger.  A Psychological Cycle , 1937 .

[53]  Gary Klein,et al.  Decision Making in Armored Platoon Command , 1990 .

[54]  David A. Cleaves,et al.  Cognitive Biases and Corrective Techniques: Proposals for Improving Elicitation Procedures for Knowledge-Based Systems , 1987, Int. J. Man Mach. Stud..

[55]  J. C. Flanagan.  Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[56]  A. L. Kidd,et al.  Knowledge acquisition for expert systems: a practical handbook , 1987 .

[57]  D. C. Miller,et al.  Evaluation of proposed control room improvements through analysis of critical operator decisions , 1981 .

[58]  Penelope M Sanderson,et al.  State-space and verbal protocol methods for studying the human operator in process control. , 1989, Ergonomics.

[59]  Philip J. Smith,et al.  A Catalog of Errors , 1992, Int. J. Man Mach. Stud..

[60]  Natalya F. Noy,et al.  The state of art in ontology design , 1997 .

[61]  Nancy J. Cooke,et al.  Varieties of knowledge elicitation techniques , 1994, Int. J. Hum. Comput. Stud..

[62]  Baruch Fischhoff,et al.  Eliciting knowledge for analytical representation , 1989, IEEE Trans. Syst. Man Cybern..

[63]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[64]  Henry H. Rueter,et al.  Extracting expertise from experts: Methods for knowledge acquisition , 1987 .

[65]  Gary Klein,et al.  Naturalistic Decision Making: Implications for Design , 1993 .

[66]  Gary Klein,et al.  Rapid Decision Making on the Fire Ground , 1986 .

[67]  J. Weitzenfeld,et al.  Analogical Reasoning as a Discovery Logic. , 1979 .

[68]  Paul J. Feltovich,et al.  A general framework for conceiving of expertise and expert systems in context , 1997 .

[69]  Joseph H. Goldberg,et al.  Battlefield Recognition of Armored Vehicles , 1995, Hum. Factors.

[70]  Clyde W. Holsapple,et al.  An exploratory study of two KA methods , 1994 .

[71]  Timothy D. Wilson,et al.  Thinking too much: introspection can reduce the quality of preferences and decisions. , 1991, Journal of personality and social psychology.

[72]  Theresa M. Mullin,et al.  Experts' estimation of uncertain quantities and its implications for knowledge acquisition , 1989, IEEE Trans. Syst. Man Cybern..

[73]  Marianne LaFrance,et al.  Excavation, Capture, Collection, and Creation: Computer Scientists' Metaphors for Eliciting Human Expertise , 1992 .

[74]  Gavriel Salvendy,et al.  Strategies and biases in human decision-making and their implications for expert systems , 1986 .

[75]  J. Cullen,et al.  The Knowledge Acquisition Bottleneck: Time for Reassessment? , 1988 .

[76]  William J. Clancey,et al.  The Epistemology of a Rule-Based Expert System - A Framework for Explanation , 1981, Artif. Intell..

[77]  N. Shadbolt,et al.  Eliciting Knowledge from Experts: A Methodological Analysis , 1995 .

[78]  Nigel Shadbolt,et al.  The empirical study of knowledge elicitation techniques , 1989, SGAR.

[79]  K. A. Ericsson,et al.  Protocol analysis: Verbal reports as data, Rev. ed. , 1993 .

[80]  Leonard Adelman,et al.  Measurement issues in knowledge engineering , 1989, IEEE Trans. Syst. Man Cybern..

[81]  Raanan Lipshitz,et al.  Why classical decision theory is an inappropriate standard for evaluating and aiding most human decision making. , 1993 .

[82]  Roberta Calderwood,et al.  Critical decision method for eliciting knowledge , 1989, IEEE Trans. Syst. Man Cybern..

[83]  Richard T. Gill,et al.  Conceptual Graph Analysis: Knowledge Acquisition for Instructional System Design , 1993 .

[84]  Gary Klein,et al.  Expert and Novice Fire Ground Command Decisions , 1987 .

[85]  S. Kay.  On the Nature of Expertise. , 1992 .

[86]  Kim J. Vicente,et al.  Supporting operator problem solving through ecological interface design , 1995, IEEE Trans. Syst. Man Cybern..

[87]  Paul E. Lehner,et al.  Expert decision-making in evolving situations , 1989, IEEE Trans. Syst. Man Cybern..

[88]  Dianne C. Berry,et al.  The problem of implicit knowledge , 1987 .