Automated Protocol Analysis

Over the past 8 years, The TextLab Research Group within the Department of Computer Science at the University of North Carolina has developed a collection of tools and techniques for recording users' interactions with graphics-based direct manipulation computer systems in machine-readable form and for automatically analyzing and displaying those data. This article describes these tools, discusses their methodological context, and considers their implications for software design and studies of human-computer interaction. Tools discussed include the following: tracking users' behaviors and producing a machine-recorded protocol at the level of users' actions, replaying users' sessions from the protocol data, modeling users' strategies using formal cognitive grammars, analyzing user sessions by parsing them with the grammars, and displaying results in visual form-both static and animated--to facilitate interpretation and understanding by researchers. These tools are placed in a methodological context by reviewing issues associated with concurrent think-aloud, keystroke, X-Windows, and video protocols; other support systems for working with these forms of protocol data are also reviewed. The discussion concludes with our reflections on the methodology and its application to computer systems and research objectives different from our own.

[1]  W. E. Mackay,et al.  EVA: an experimental video annotator for symbolic analysis of video data , 1989, SGCH.

[2]  R. H. Trigg,et al.  Computer support for transcribing recorded activity , 1989, SGCH.

[3]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[4]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[5]  Jeffrey J. Hendrickson,et al.  Integrated data capture and analysis tools for research and testing on graphical user interfaces , 1992, CHI.

[6]  Ii John Quillian Walker.  Automated analysis of computer-generated software usage protocols: an exploratory study , 1992 .

[7]  Carolanne Fisher.  Advancing the study of programming with computer-aided protocol analysis , 1987 .

[8]  Antonio C. Siochi,et al.  Computer analysis of user interfaces based on repetition in transcripts of user sessions , 1991, TOIS.

[9]  Jay David Bolter,et al.  WE: A Writing Environment for Professionals, , 1986 .

[10]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[11]  James D. Foley,et al.  The art of natural graphic man—Machine conversation , 1974 .

[12]  Marcy Lansman,et al.  Designing theory-based systems: a case study , 1992, CHI.

[13]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[14]  John B. Smith,et al.  Collective Intelligence in Computer-Based Collaboration , 1994 .

[15]  William A. Woods,et al.  Computational Linguistics Transition Network Grammars for Natural Language Analysis , 2022 .

[16]  A. Newell.  Unified Theories of Cognition , 1990 .

[17]  Pedro Sánchez,et al.  Collaborative technology and group process feedback: their impact on interactive sequences in meetings , 1990, CSCW '90.

[18]  Hayes.  identifying the organization of wi iiing processes , 1980 .

[19]  Allen Newell,et al.  PAS-II: An Interactive Task-Free Version of an Automatic Protocol Analysis System , 1973, IEEE Transactions on Computers.

[20]  Lucy Suchman,et al.  Understanding practice: video as a medium for reflection and design (excerpt) , 1992 .

[21]  John B. Smith,et al.  A hypertext writing environment and its cognitive basis (panel session) , 1987, Hypertext.

[22]  R. Guindon,et al.  Video: Data for studying human-computer interaction , 1988, CHI '88.

[23]  Judith S. Olson,et al.  User‐centered design of collaboration technology , 1991 .

[24]  R. Bales,et al.  Symlog, A System for the Multiple Level Observation of Groups , 1979 .

[25]  Janet H. Walker,et al.  A case study of using a manual online , 1991 .

[26]  Phyllis Reisner,et al.  Formal Grammar and Human Factors Design of an Interactive Graphics System , 1981, IEEE Transactions on Software Engineering.

[27]  Marcy Lansman,et al.  A Cognitive Basis for a Computer Writing Environment , 1990 .

[28]  Allen Newell,et al.  Protocol Analysis as a Task for Artificial Intelligence , 1971, IJCAI.

[29]  Allen Newell,et al.  Human Problem Solving. , 1973 .