Thinking aloud as a method for testing the usability of Websites: the influence of task variation on the evaluation of hypertext

In the usability testing of Web sites, thinking aloud is a frequently-used method. A fundamental discussion, however, about the relation between the use of different variants of thinking aloud and the evaluation goals for this specific medium is still lacking. To lay a foundation for this discussion, I analyzed the results of three usability studies in which different thinking-aloud tasks were used: a simple searching task, an application task and a prediction task. In the task setting, the profile of the Web surfer, the communication goal of the Web site and other quality aspects are taken into account. The qualitative analysis of these studies shows that the task variation has some influence on the results of usability testing and that, consequently, tasks should be matched with the evaluation goals put forward.

[1]  Carolyn Snyder,et al.  Web Site Usability: A Designer's Guide , 1997 .

[2]  Janice Redish,et al.  User and task analysis for interface design , 1998 .

[3]  Karen A. Schriver.  Dynamics in Document Design: Creating Text for Readers , 1996 .

[4]  Peter Jan Schellens,et al.  Reader-Focused Text Evaluation , 1997 .

[5]  Eric J. Johnson,et al.  The validity of verbal protocols , 1989, Memory & cognition.

[6]  JoAnn T. Hackos,et al.  Standards for Online Communication: Publishing Information for the Internet/World Wide Web/Help Systems/Corporate Intranets , 1997 .

[7]  Sarah Ellen Ransdell.  GENERATING THINKING-ALOUD PROTOCOLS: IMPACT ON THE NARRATIVE WRITING OF COLLEGE STUDENTS , 1995 .

[8]  Huub van den Bergh,et al.  Effects of thinking aloud on writing processes , 1996 .

[9]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[10]  Tom Carey,et al.  Labeled, typed links as cues when reading hypertext documents , 1996 .

[11]  Steve Muylle,et al.  A grounded theory of World Wide Web search behaviour , 1999 .

[12]  Susan H. Gray,et al.  Using protocol analyses and drawings to study mental model construction during hypertext navigation , 1990, Int. J. Hum. Comput. Interact..

[13]  Karen A. Schriver.  Evaluating Text Quality: The Continuum from Text-Focused to Reader-Focused Methods. Technical Report No. 41. , 1989 .

[14]  Peter Smagorinsky.  Speaking about writing : reflections on research methodology , 1994 .

[15]  Thomas S. Tullis,et al.  A "bag of tricks" for web usability , 2000, CHI Extended Abstracts.

[16]  Peter Smagorinsky,et al.  The Reliability and Validity of Protocol Analysis , 1989 .

[17]  Jared M. Spool,et al.  Measuring web-site usability , 2000, CHI Extended Abstracts.

[18]  Luuk van Waes.  Evaluating on-line and off-line searching behavior using thinking-aloud protocols to detect navigation barriers , 1998, SIGDOC '98.