A Knowledge Elicitation Technique for Educational Development: The Critical Decision Method.

The Critical Decision Method (CDM) is a structured interview method for eliciting expert knowledge. The method was used in a study of computer programmers who were experts at debugging comple7 computer systems. Fifteen programmers who were identified by their supervisors as experts were asked to describe an experience in which their expertise made a difference. The CDM interviewer used a set of questions to elaborate the programmers' responses. Two non-expert programmers were interviewed for craparison. Each subject told the story of a debugging experience four times with varying degrees of intervention by the interviewers. Data were validated by a panel of eight subjects who reviewed the findings of the interviews. The results show that the CDM facilitated the identification and description of expert skills (critical knowledge) as we7l as resources and heuristics for selection and use of skills and resources by the experts. The storJes produced have the potential to enhance training materials for other programmers. A 32-item list of references is included. (SLD) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. *********************************************************************** A Knowledge Elicitation Technique for Educational Development: The Critical Decision Method Jared Freeman, Teachers College, Columbia University Julian Weitzenfeld, JS. Weitzenfeld Consulting, East Windsor, NJ Gary Klein, Klein Associates, Yellow Springs, CH Tom Riedl, Ried; Consul New York, NY U.& DEPARTMENT OF EDUCATION Office or Erhoc ahem. Researcherto improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) 14ms document has Dean reptDouceo Is rimmed horn the person or orgentzsbon Orripna Itng r Min Or changes have been made to tmorove teorcduchon oughts Points la We* Op.nion I Stilted in this cloc union! do rlOt necessarlho ropreserrt &kW OE RI Dosmon or Doncv ting, John Musa, AT&T Dell Laboratories, Murray Hill, NJ Paper presented at the annual conference of the American Educational Research Association Chicago, Illinois April 6, 1991 Abstract "PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY IRAEJ r4EEILI 1171)PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY IRAEJ r4EEILI 1171) TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)." The Critical Decision Method (CDM) is a structured interview method for eliciting expert knowledge. The method was employed in a study of computer programmers who are expert at debugging complex computer systems. The role of the CDM in that study is described and the method is critiqued.

[1]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[2]  J. C. Flanagan.  Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[3]  Robert E. Kraut,et al.  Listener Responsiveness and the Coordination of Conversation , 1982 .

[4]  Ruven E. Brooks,et al.  Studying programmer behavior experimentally: the problems of proper methodology , 1980, CACM.

[5]  N. Pennington,et al.  Evidence evaluation in complex decision making. , 1986 .

[6]  P. Thorndyke.  Cognitive structures in comprehension and memory of narrative discourse , 1977, Cognitive Psychology.

[7]  Marianne LaFrance,et al.  The quality of expertise: implications of expert-novice differences for knowledge acquisition , 1989, SGAR.

[8]  Walter Kintsch,et al.  Comprehension and recall of text as a function of content variables , 1975 .

[9]  B. A. Sheil,et al.  The Psychological Study of Programming , 1981, CSUR.

[10]  John A. Gould,et al.  Some Psychological Evidence on How People Debug Computer Programs , 1975, Int. J. Man Mach. Stud..

[11]  Herbert P. Ginsburg,et al.  Children's Arithmetic: How They Learn It and How You Teach It , 1977 .

[12]  Richard C. Anderson,et al.  A Schema-Theoretic View of Basic Processes in Reading Comprehension. Technical Report No. 306. , 1988 .

[13]  John B. Black,et al.  Scripts in memory for text , 1979, Cognitive Psychology.

[14]  Steven H. Lewis,et al.  Listener Responsiveness and the Coordination of Conversation , 1982 .

[15]  G. A. Klein,et al.  Knowledge engineering : beyond expert systems , 1990 .

[16]  A. Tversky.  Elimination by aspects: A theory of choice. , 1972 .

[17]  E. Tulving,et al.  Episodic and semantic memory , 1972 .

[18]  A. Tversky,et al.  The simulation heuristic , 1982 .

[19]  Gary Klein,et al.  Rapid Decision Making on the Fire Ground , 1986 .

[20]  Gary Klein,et al.  Use of a prediction paradigm to evaluate proficient decision making , 1989 .

[21]  Beth Crandall,et al.  A comparative study of think-aloud and critical decision knowledge elicitation methods , 1989, SGAR.

[22]  Herbert P. Ginsburg,et al.  The development of mathematical thinking , 1983 .

[23]  Roberta Calderwood,et al.  Critical decision method for eliciting knowledge , 1989, IEEE Trans. Syst. Man Cybern..

[24]  Marianne LaFrance,et al.  The Knowledge Acquisition Grid: A Method for Training Knowledge Engineers , 1987, Int. J. Man Mach. Stud..

[25]  John R. Anderson.  Cognitive Psychology and Its Implications , 1980 .

[26]  Robert M. Krauss,et al.  The Role of the Listener: Addressee Influences on Message Formulation , 1987 .

[27]  H. H. Clark,et al.  References in Conversation Between Experts and Novices , 1987 .