Retrospective protocols in usability testing: a comparison of Post-session RTA versus Post-task RTA reports

ABSTRACT We present the results of a study that compared two placements of the Retrospective Think-aloud (RTA): A Post-session RTA where the think-aloud occurs after all tasks are complete, and a Post-task RTA where the think-aloud is elicited after each task. Data from task performance and verbal measures were collected from 24 participants. The results suggest that in terms of task performance, participants in the Post-session RTA condition performed tasks faster, with fewer errors and fewer clicks than in the Post-task RTA condition. In terms of utterances, participants in the Post-task RTA condition produced significantly more utterances that explained actions, expectations and procedural descriptions than in the Post-session RTA condition.

[1]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[2]  Qingxin Shi,et al.  A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests , 2008, NordiCHI.

[3]  Kasper Hornbæk,et al.  Cooperative usability testing: complementing usability tests with user-supported interpretation sessions , 2005, CHI Extended Abstracts.

[4]  Ronald P. Leow.  MODELS, ATTENTION, AND AWARENESS IN SLA , 2002, Studies in Second Language Acquisition.

[5]  Kasper Hornbæk,et al.  Work-domain knowledge in usability evaluation: Experiences with Cooperative Usability Testing , 2010, J. Syst. Softw..

[6]  David W. Biers,et al.  Retrospective versus Concurrent Thinking-Out-Loud in Usability Testing , 1993 .

[7]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[8]  Tingting Zhao,et al.  Dual Verbal Elicitation: The Complementary Use of Concurrent and Retrospective Reporting Within a Usability Test , 2013, Int. J. Hum. Comput. Interact..

[9]  Zhiwei Guan,et al.  The validity of the stimulated retrospective think-aloud method as measured by eye tracking , 2006, CHI.

[10]  Mark C. Fox,et al.  Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. , 2011, Psychological bulletin.

[11]  Yvonne Kammerer,et al.  Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data , 2011 .

[12]  Morten Hertzum,et al.  Thinking Aloud in the Presence of Interruptions and Time Constraints , 2013, Int. J. Hum. Comput. Interact..

[13]  Peter Jan Schellens,et al.  Evaluation of an Informational Web Site: Three Variants of the Think-aloud Method Compared , 2007 .

[14]  Tingting Zhao,et al.  Keep talking: an analysis of participant utterances gathered using two concurrent think-aloud methods , 2010, NordiCHI.

[15]  Mansour Rahimi,et al.  Concurrent and Retrospective Verbal Protocols in Usability Testing: Is there Value Added in Collecting Both? , 1995 .

[16]  Helen Petrie,et al.  The effect of global instructions on think-aloud testing , 2013, CHI.

[17]  Elizabeth D. Murphy,et al.  Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability , 2010, CHI.

[18]  Linden J. Ball,et al.  Cueing retrospective verbal reports in usability testing through eye-movement replay , 2007, BCS HCI.

[19]  Menno D. T. de Jong,et al.  Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue , 2003, Behav. Inf. Technol..

[20]  Shu Ching Yang,et al.  Reconceptualizing think-aloud methodology: refining the encoding and categorizing techniques via contextualized perspectives , 2003, Comput. Hum. Behav..

[21]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.

[22]  Jean-Paul Dionne,et al.  Accessing Problem-Solving Strategy Knowledge: The Complementary Use of Concurrent Verbal Protocols and Retrospective Debriefing. , 2000 .

[23]  Morten Hertzum,et al.  What Do Thinking-Aloud Participants Say? A Comparison of Moderated and Unmoderated Usability Sessions , 2015, Int. J. Hum. Comput. Interact..

[24]  P. Hancock,et al.  How cognitive load affects duration judgments: A meta-analytic review. , 2010, Acta psychologica.

[25]  Menno D. T. de Jong,et al.  Evaluating municipal websites: A methodological comparison of three think-aloud variants , 2009, Gov. Inf. Q..

[26]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[27]  Morten Hertzum,et al.  Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? , 2009, Behav. Inf. Technol..

[28]  Tingting Zhao,et al.  Look Who's Talking: Evaluating the Utility of Interventions During an Interactive Think-Aloud , 2016, Interact. Comput..

[29]  Tingting Zhao,et al.  Exploring Think-Alouds in Usability Testing: An International Survey , 2012, IEEE Transactions on Professional Communication.

[30]  Menno D. T. de Jong,et al.  Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison , 2004, Interact. Comput..

[31]  Leo Lentz,et al.  Retrospective think-aloud method: using eye movements as an extra cue for participants' verbalizations , 2011, CHI.

[32]  Victoria A. Bowers.  Concurrent versus Retrospective Verbal Protocol for Comparing Window Usability , 1990 .

[33]  Jacob O. Wobbrock,et al.  Understanding usability practices in complex domains , 2010, CHI.

[34]  Karen A. Gresty,et al.  Reflecting on the think-aloud method for evaluating e-learning , 2006, Br. J. Educ. Technol..

[35]  Tingting Zhao,et al.  The impact of two different think-aloud instructions in a usability test: a case of just following orders? , 2014, Behav. Inf. Technol..