Verbal reports and domain-specific knowledge: a comparison between collegial and retrospective verbalisation

One way to investigate and account for the role of experience in dynamic decision-making tasks is to use a knowledge elicitation method, for example verbal protocols. Recently, methods for verbalisation by other subjects have been suggested as a way to elicit more information on thinking. One of these methods is here investigated in a quasi-experimental study on highly experienced and skilled train dispatchers. Collegial verbalisation is based on the procedure of videotaping practitioners while they perform their normal work tasks in their normal work setting. This is followed up by having a close colleague of the practitioner watch the video recordings and verbalise. The general hypothesis that verbal reports based on collegial verbalisations can provide protocol data that are close to the structure and content of verbal reports based on retrospective verbalisations was investigated. From this systematic comparison it is concluded that collegial verbalisations produce verbal protocols that are close to the retrospective verbal protocols on protocol and topic levels, but not on statement level. From this study, we conclude that the collegial protocols can be used as an independent source of data. It seems possible for a colleague to report verbally on a practitioners’ observable behaviour in the same way as when the practitioner is doing a retrospective verbalisation and it may also be possible for a colleague to explain some of the non-observable behaviour of the practitioner.

[1]  Richard W. Pew,et al.  Evaluating Nuclear Control Room Improvements Through Analysis of Critical Operator Decisions , 1981 .

[2]  K. J. Vicente,et al.  Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work , 1999 .

[3]  Pallab Paul,et al.  A comparison of concurrent and retrospective verbal protocol analysis. , 2000, The American journal of psychology.

[4]  K. R. Hammond.  Judgments Under Stress , 1999 .

[5]  S. Sonnentag.  Expertise in professional software design: a process study. , 1998, The Journal of applied psychology.

[6]  Maarten van Someren,et al.  The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes , 1994 .

[7]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[8]  Bengt Sandblad,et al.  Control strategies for managing train traffic : Difficulties today and visions for the future , 2006 .

[9]  N. Pennington,et al.  Transfer of Training Between Cognitive Subskills: Is Knowledge Use Specific? , 1995, Cognitive Psychology.

[10]  Matthew W. Lewis,et al.  Self-Explonations: How Students Study and Use Examples in Learning to Solve Problems , 1989, Cogn. Sci..

[11]  J. Shaoul.  Human Error , 1973, Nature.

[12]  Nancy J. Cooke,et al.  Knowledge Elicitation , 2003 .

[13]  Johan Wikström,et al.  Future train traffic control: control by re-planning , 2005, Cognition, Technology & Work.

[14]  D. Cannon,et al.  Correspondence between MMPI PTSD measures and clinical diagnosis. , 1987, Journal of personality assessment.

[15]  Ann L. Brown,et al.  Training in Self-Explanation and Self-Regulation Strategies: Investigating the Effects of Knowledge Acquisition Activities on Problem Solving , 1995 .

[16]  S. Dumais,et al.  Handbook of applied cognition , 2007 .

[17]  Elliot Soloway,et al.  A cognitive analysis of a code inspection , 1987 .

[18]  Françoise Détienne,et al.  On the use of functional and interactional approaches for the analysis of technical review meetings , 2000, PPIG.

[19]  Gerrit C. van der Veer,et al.  Mental Representations of Computer Languages — a Lesson from Practice , 1993 .

[20]  William B. Swann,et al.  The Cognitive-Affective Crossfire , 1987 .

[21]  Jens Rasmussen,et al.  Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[22]  Michael D. McNeese.  How video informs cognitive systems engineering: making experience count , 2004, Cognition, Technology & Work.

[23]  Steve Parkes,et al.  Trends in the use of verbal protocol analysis in software engineering research , 2003, Behav. Inf. Technol..

[24]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[25]  K. A. Ericsson.  Protocol Analysis in Psychology , 2001 .

[26]  Nancy Pennington,et al.  Comprehension strategies in programming , 1987 .

[27]  B. Brehmer.  Dynamic decision making: human control of complex systems. , 1992, Acta psychologica.

[28]  Vairam Arunachalam,et al.  Cognitive processes in program comprehension: An empirical analysis in the Context of software reengineering , 1996, J. Syst. Softw..

[29]  Michelene T. H. Chi,et al.  Self-Explanations: How Students Study and Use Examples in Learning To Solve Problems. Technical Report No. 9. , 1987 .

[30]  Neville A. Stanton,et al.  Observing the observer: non-intrusive verbalisations using the Concurrent Observer Narrative Technique , 2011, Cognition, Technology & Work.

[31]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[32]  Elliot Soloway,et al.  Empirical Studies of Programmers: Second Workshop , 1991 .

[33]  G. Klein,et al.  Decision Making in Action: Models and Methods , 1993 .

[34]  Nadine B. Sarter,et al.  How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control , 1995, Hum. Factors.

[35]  Eileen Cormier,et al.  Cognitions associated with nurse performance: a comparison of concurrent and retrospective verbal reports of nurse performance in a simulated task environment. , 2010, International journal of nursing studies.

[36]  Jens Rasmussen,et al.  Virtual ecology of work , 1995 .

[37]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[38]  Iris Vessey,et al.  Expertise in Debugging Computer Programs: A Process Analysis , 1984, Int. J. Man Mach. Stud..

[39]  Giuliana Dettori,et al.  Cognitive Models and Intelligent Environments for Learning Programming , 1993, NATO ASI Series.

[40]  Anders Jansson,et al.  Bridging the gap between analysis and design: improving existing driver interfaces with tools from the framework of cognitive work analysis , 2005, Cognition, Technology & Work.

[41]  Nicholas P. Vitalari,et al.  Differences Between Novice and Expert Systems Analysts: What Do We Know and What Do We Do? , 1998, J. Manag. Inf. Syst..

[42]  G. Klein,et al.  A recognition-primed decision (RPD) model of rapid decision making. , 1993 .

[43]  Emily S. Patterson,et al.  Elicitation by critiquing as a cognitive task analysis methodology , 2006, Cognition, Technology & Work.

[44]  Lisanne Bainbridge,et al.  Verbal reports as evidence of the process operator's knowledge , 1999, Int. J. Hum. Comput. Stud..

[45]  F. Gibbons.  Self-attention and self-report: The "veridicality" hypothesis. , 1983, Journal of personality.

[46]  Anders Jansson,et al.  Collegial verbalisation – a case study on a new method on information acquisition , 2007, Behav. Inf. Technol..

[47]  M. Polanyi.  Personal Knowledge: Towards a post-critical philosophy , 1959 .

[48]  Benjamin Kuipers,et al.  How to Discover a Knowledge Representation for Causal Reasoning by Studying an Expert Physician , 1983, IJCAI.

[49]  Jim McLennan,et al.  Effects of Concurrent Verbalization on a Time-Critical, Dynamic Decision-Making Task , 2000, The Journal of general psychology.

[50]  Robert J. Crutcher,et al.  INTROSPECTION AND VERBAL REPORTS ON COGNITIVE PROCESSES-TWO APPROACHES TO THE STUDY OF THINKING: A RESPONSE TO HOWE* , 1991 .