Gaze Path Stimulation in Retrospective Think-Aloud

For a long time, eye tracking has been thought of as a promising method for usability testing. During the last couple of years, eye tracking has finally started to live up to these expectations, at least in terms of its use in usability laboratories. We know that the user’s gaze path can reveal usability issues that would otherwise go unnoticed, but a common understanding of how best to make use of eye movement data has not been reached. Many usability practitioners seem to have intuitively started to use gaze path replays to stimulate recall for retrospective walk through of the usability test. We review the research on thinkaloud protocols in usability testing and the use of eye tracking in the context of usability evaluation. We also report our own experiment in which we compared the standard, concurrent think-aloud method with the gaze path stimulated retrospective think-aloud method. Our results suggest that the gaze path stimulated retrospective think-aloud method produces more verbal data, and that the data are more informative and of better quality as the drawbacks of concurrent think-aloud have been avoided.

[1]  Eric J. Johnson,et al.  The validity of verbal protocols , 1989, Memory & cognition.

[2]  Yan Xiao,et al.  Using Eye-Tracking Video Data to Augment Knowledge Elicitation in Cognitive Task Analysis , 2001 .

[3]  D. S. Wooding,et al.  Fixation maps: quantifying eye-movement traces , 2002, ETRA.

[4]  Rico Fischer,et al.  Eye tracking and its application in usability and media research , 2003, MMI Interakt..

[5]  Tao Lin,et al.  Evaluating usability based on multimodal information: an empirical study , 2006, ICMI '06.

[6]  Vanessa Evers.  Cross-cultural applicability of user evaluation methods: a case study amongst Japanese, North-American, English and Dutch users , 2002, CHI Extended Abstracts.

[7]  Timothy D. Wilson,et al.  The Proper Protocol: Validity and Completeness of Verbal Reports , 1994 .

[8]  John Paulin Hansen,et al.  New technological windows into mind: there is more in eyes and brains for human-computer interaction , 1996, CHI.

[9]  Miranda G. Capra.  Contemporaneous versus Retrospective User-Reported Critical Incidents in Usability Evaluation , 2002 .

[10]  Pallab Paul,et al.  A comparison of concurrent and retrospective verbal protocol analysis. , 2000, The American journal of psychology.

[11]  Jennifer L. Branch.  Junior high students and Think Alouds generating information-seeking process data using concurrent verbal protocols , 2001 .

[12]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[13]  Stephanie Wilson,et al.  Eye-Tracking Data Identifying Web Usability Problems from , 2007 .

[14]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[15]  Janet Finlay,et al.  Getting a measure of satisfaction from eyetracking in practice , 2006, CHI EA '06.

[16]  Caroline Jarrett.  Label placement in forms: what's best? , 2008 .

[17]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[18]  F. Paas,et al.  Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting. , 2005, Journal of experimental psychology. Applied.

[19]  John L. Arnott,et al.  Methods for human – computer interaction research with older people , 2007, Behav. Inf. Technol..

[20]  Ken-ichi Matsumoto,et al.  Detecting low usability web pages using quantitative data of users' behavior , 2006, ICSE.

[21]  Elizabeth D. Murphy,et al.  Usability Testing of an Internet Form for the 2004 Overseas Enumeration Test: Iterative Testing Using Think-Aloud and Retrospective Report Methods , 2004 .

[22]  Joseph H. Goldberg,et al.  Chapter 23 – Eye Tracking in Usability Evaluation: A Practitioner's Guide , 2003 .

[23]  Linden J. Ball,et al.  Applying the post-experience eye-tracked protocol (PEEP) method in usability testing. , 2006 .

[24]  Zhiwei Guan,et al.  The validity of the stimulated retrospective think-aloud method as measured by eye tracking , 2006, CHI.

[25]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[26]  Joseph H. Goldberg,et al.  Eye tracking in web search tasks: design implications , 2002, ETRA.

[27]  Jennifer L Branch.  Investigating the Information-Seeking Processes of Adolescents: The Value of Using Think Alouds and Think Afters , 2000 .

[28]  L. Cooke,et al.  Using eye tracking to address limitations in think-aloud protocol , 2005, IPCC 2005. Proceedings. International Professional Communication Conference, 2005..

[29]  M. W. van Someren,et al.  The think aloud method: a practical approach to modelling cognitive processes , 1994 .

[30]  Gerhard P. Deffner,et al.  Evaluation of Concurrent Thinking Aloud Using Eye-tracking Data , 1990 .

[31]  J. Sandberg,et al.  The think aloud method , 1994 .

[32]  Merja Lehtinen,et al.  A gaze path cued retrospective thinking aloud technique in usability testing , 2007 .

[33]  Agnieszka Bojko,et al.  Using eye tracking to compare web page designs: a case study , 2006 .

[34]  Stephanie Wilson,et al.  Identifying web usability problems from eye-tracking data , 2007, BCS HCI.

[35]  K. Taylor,et al.  Accessing Problem-Solving Strategy Knowledge: The Complementary Use of Concurrent Verbal Protocols and Retrospective Debriefing. , 2000 .

[36]  André P. Calitz,et al.  The added value of eye tracking in the usability evaluation of a network management tool , 2005 .

[37]  Linden J. Ball,et al.  Cueing retrospective verbal reports in usability testing through eye-movement replay , 2007, BCS HCI.

[38]  Menno D. T. de Jong,et al.  Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue , 2003, Behav. Inf. Technol..

[39]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[40]  J. Edward Russo,et al.  A software system for the collection of retrospective protocols prompted by eye fixations , 1979 .

[41]  Susan M. Denning,et al.  The Value of Thinking-Aloud Protocols in Industry: A Case Study at Microsoft Corporation , 1990 .

[42]  J. P. Hansen.  The use of eye mark recordings to support verbal retrospection in software testing , 1991 .

[43]  Linden J. Ball,et al.  Eye tracking in HCI and usability research. , 2006 .

[44]  Victoria A. Bowers.  Concurrent versus Retrospective Verbal Protocol for Comparing Window Usability , 1990 .

[45]  Janni Nielsen,et al.  Getting access to what goes on in people's heads?: reflections on the think-aloud technique , 2002, NordiCHI '02.

[46]  Maria Ebling,et al.  On the contributions of different empirical data in usability testing , 2000, DIS '00.

[47]  J. Edward Russo,et al.  Eye Fixations Can Save the World: a Critical Evaluation and a Comparison Between Eye Fixations and Other Information Processing Methodologies , 1978 .

[48]  Robert J. K. Jacob,et al.  Eye tracking in human-computer interaction and usability research : Ready to deliver the promises , 2002 .

[49]  Heejung S. Kim,et al.  We talk, therefore we think? A cultural analysis of the effect of talking on thinking. , 2002, Journal of personality and social psychology.

[50]  P. Yammiyavar,et al.  Influence of Cultural Background on Non-verbal Communication in a Usability Testing Situation , 2008 .

[51]  John Paulin Hansen,et al.  Do we need eye trackers to tell where people look? , 2006, CHI Extended Abstracts.

[52]  Corina Sas,et al.  People and Computers XXI - HCI... but not as we know it: Proceedings of HCI. , 2007 .

[53]  Terry C. Lansdown,et al.  The mind's eye: cognitive and applied aspects of eye movement research , 2005 .