The impact of two different think-aloud instructions in a usability test: a case of just following orders?

The instructions used in think-aloud studies can range from a simple request to think-aloud, to an explicit instruction to include certain types of content. The present study compared two think-aloud instructions: the classic neutral think-aloud instruction and an explicit instruction requesting explanations and content that is relevant to the user experience. Data from task performance, mental workload, think-aloud protocols and usability problems were collected from 16 participants, equally distributed between the two think-aloud instruction conditions. No differences were found in task performance, however, participants in the explicit instruction condition reported higher mental workload and a focus on finding interface problems. The explicit instruction condition also yielded more utterances about the user experience, expectations and explanations of behaviour than the neutral condition. An analysis of the resultant usability problems revealed that the explicit instruction led to a larger number of dialogue, navigation, layout and functionality problems, but that the problems which were unique to this condition were, in the main, at a low level of severity.

[1]  Eric J. Johnson,et al.  The validity of verbal protocols , 1989, Memory & cognition.

[2]  E. Krahmer,et al.  Thinking about thinking aloud: a comparison of two verbal protocols for usability testing , 2004, IEEE Transactions on Professional Communication.

[3]  Wolmet Barendregt,et al.  Predicting effectiveness of children participants in user testing based on personality characteristics , 2007, Behav. Inf. Technol..

[4]  Gilbert Cockton,et al.  A Framework for Usability Problem Extraction , 1999, INTERACT.

[5]  Malcolm P. Atkinson,et al.  Comparison of evaluation methods using structured usability problem reports , 1997, Behav. Inf. Technol..

[6]  Qingxin Shi,et al.  A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests , 2008, NordiCHI.

[7]  Timothy D. Wilson,et al.  The Proper Protocol: Validity and Completeness of Verbal Reports , 1994 .

[8]  Kasper Hornbæk,et al.  Working Together to Improve Usability : Challenges and Best Practices , 2008 .

[9]  Gitte Lindgaard,et al.  Usability testing: what have we overlooked? , 2007, CHI.

[10]  Elizabeth D. Murphy,et al.  Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability , 2010, CHI.

[11]  M. Chi.  Quantifying Qualitative Analyses of Verbal Data: A Practical Guide , 1997 .

[12]  R. T. Evans.  Usability testing and research , 2002 .

[13]  Yvonne Kammerer,et al.  Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data , 2011 .

[14]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[15]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[16]  Joseph S. Dumas,et al.  Moderating Usability Tests: Principles and Practices for Interacting: Principles and Practices for Interacting , 2008 .

[17]  Jan Gulliksen,et al.  Usability professionals - current practices and future development , 2006, Interact. Comput..

[18]  Arnold P. O. S. Vermeeren,et al.  Managing the 'Evaluator Effect' in User Testing , 2003, INTERACT.

[19]  M.D.T. de Jong,et al.  Constructive Interaction: An Analysis of Verbal Interaction in a Usability Setting , 2006, IEEE Transactions on Professional Communication.

[20]  Michelene T. H. Chi,et al.  Eliciting Self-Explanations Improves Understanding , 1994, Cogn. Sci..

[21]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[22]  Huib de Ridder,et al.  DEVAN: A tool for detailed video analysis of user test data , 2002, Behav. Inf. Technol..

[23]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[24]  Tingting Zhao,et al.  Keep talking: an analysis of participant utterances gathered using two concurrent think-aloud methods , 2010, NordiCHI.

[25]  Richard B. Wright,et al.  Method Bias and Concurrent Verbal Protocol in Software Usability Testing , 1992 .

[26]  Maria Ebling,et al.  On the contributions of different empirical data in usability testing , 2000, DIS '00.

[27]  Kasper Hornbæk,et al.  Comparison of techniques for matching of usability problem descriptions , 2008, Interact. Comput..

[28]  Wolmet Barendregt,et al.  Identifying usability and fun problems in a computer game during first use and after some practice , 2006, Int. J. Hum. Comput. Stud..

[29]  Lynne Cooke,et al.  Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach , 2010, IEEE Transactions on Professional Communication.

[30]  Kasper Hornbæk,et al.  Dogmas in the assessment of usability evaluation methods , 2010, Behav. Inf. Technol..

[31]  Mark C. Fox,et al.  Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. , 2011, Psychological bulletin.

[32]  Martin C. Maguire,et al.  Context of Use within usability activities , 2001, Int. J. Hum. Comput. Stud..

[33]  Tingting Zhao,et al.  Exploring Think-Alouds in Usability Testing: An International Survey , 2012, IEEE Transactions on Professional Communication.

[34]  Carol M. Barnum.  Usability Testing Essentials: Ready, Set...Test! , 2010 .

[35]  Morten Hertzum,et al.  The evaluator effect in usability tests , 1998, CHI Conference Summary.

[36]  Victoria A. Bowers.  Concurrent versus Retrospective Verbal Protocol for Comparing Window Usability , 1990 .

[37]  Jacob O. Wobbrock,et al.  Understanding usability practices in complex domains , 2010, CHI.

[38]  Jonathan W. Schooler,et al.  Why do words hurt? Content, process, and criterion shift accounts of verbal overshadowing , 2008 .

[39]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[40]  Susanne Friese,et al.  Qualitative Data Analysis with ATLAS.ti , 2012 .

[41]  Morten Hertzum,et al.  Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? , 2009, Behav. Inf. Technol..

[42]  Jonathan W Schooler,et al.  Introspecting in the spirit of William James: comment on Fox, Ericsson, and Best (2011). , 2011, Psychological bulletin.

[43]  Ann Blandford,et al.  This is what i'm doing and why: reflections on a think-aloud study of dl users' information behaviour , 2010, JCDL '10.

[44]  Shu Ching Yang,et al.  Reconceptualizing think-aloud methodology: refining the encoding and categorizing techniques via contextualized perspectives , 2003, Comput. Hum. Behav..

[45]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.