"Thinking About Thinking Aloud": An Investigation of Think-Aloud Methods in Usability Testing

Usability has become an imperative aspect of survival on the web, thus, it has always been considered as a crucial aspect of web design. This paper presents the results of a study that compared two think-aloud usability testing methods: the concurrent think-aloud and the retrospective think-aloud methods. Data from task performance, testing experience, and usability problems were collected from 40 participants equally distributed between the two think-aloud conditions. The results found that while the thinking aloud method had no impact on task performance and participants testing experience, participants using the concurrent think-aloud method detected a larger number of minor problems with the test interface than participants using the retrospective think-aloud method. These findings suggest a reason for preferring the concurrent think-aloud method to the retrospective one.

[1]  Jakob Nielsen,et al.  Determining Usability Test Sample Size , 2006 .

[2]  Miles MacLeod,et al.  Usability measurement in context , 1994, Behav. Inf. Technol..

[3]  Klaus Kaasgaard,et al.  Comparative usability evaluation , 2004, Behav. Inf. Technol..

[4]  G. Winship.  Case Study Research in Practice , 2010 .

[5]  Tingting Zhao,et al.  Keep talking: an analysis of participant utterances gathered using two concurrent think-aloud methods , 2010, NordiCHI.

[6]  Mikael B. Skov,et al.  Comparison of think-aloud and constructive interaction in usability testing with children , 2005, IDC '05.

[7]  J. Watson,et al.  Conditioned emotional reactions , 1920 .

[8]  Jakob Nielsen,et al.  Guerrilla HCI: using discount usability engineering to penetrate the intimidation barrier , 1994 .

[9]  Samuel Ssemugabi,et al.  Usability testing of e-learning: an approach incorporating co-discovery and think-aloud , 2009 .

[10]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[11]  Marjan Sarshar,et al.  Quantitative and qualitative research in the built environment: application of “mixed” research approach , 2002 .

[12]  Wolmet Barendregt,et al.  Identifying usability and fun problems in a computer game during first use and after some practice , 2006, Int. J. Hum. Comput. Stud..

[13]  James R. Lewis,et al.  Sample sizes for usability tests: mostly math, not magic , 2006, INTR.

[14]  Lynne Cooke,et al.  Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach , 2010, IEEE Transactions on Professional Communication.

[15]  David R. Krathwohl,et al.  Methods of Educational and Social Science Research: An Integrated Approach , 1992 .

[16]  Kasper Hornbæk,et al.  Dogmas in the assessment of usability evaluation methods , 2010, Behav. Inf. Technol..

[17]  Emerson Abraham Jackson.  Media Review: Practical Research Methods: A User-Friendly Guide to Mastering Research Techniques and Projects , 2017 .

[18]  Kasper Hornbæk,et al.  Working Together to Improve Usability : Challenges and Best Practices , 2008 .

[19]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[20]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[21]  L NormanKent,et al.  Levels of automation and user participation in usability testing , 2006 .

[22]  R. T. Evans.  Usability testing and research , 2002 .

[23]  Monique W. M. Jaspers,et al.  Cognitive Evaluation of a Physician Data Query Tool for a National ICU Registry: Comparing Two Think Aloud Variants and Their Application in Redesign , 2010, MedInfo.

[24]  O. J. Dunn.  Multiple Comparisons Using Rank Sums , 1964 .

[25]  Jürgen Sauer,et al.  The influence of user expertise and prototype fidelity in usability tests. , 2010, Applied ergonomics.

[26]  Arnold P. O. S. Vermeeren,et al.  Managing the 'Evaluator Effect' in User Testing , 2003, INTERACT.

[27]  Ann Blandford,et al.  This is what I'm doing and why: Methodological reflections on a naturalistic think-aloud study of interactive information behaviour , 2011, Inf. Process. Manag..

[28]  Ebba Þóra Hvannberg,et al.  Consolidating usability problems with novice evaluators , 2008, NordiCHI.

[29]  Mark C. Fox,et al.  Thinking aloud is not a form of introspection but a qualitatively different methodology: reply to Schooler (2011). , 2011, Psychological bulletin.

[30]  M. Furlong,et al.  Eight Was Not Enough , 2009 .

[31]  Herre van Oostendorp,et al.  Learning by exploration: Thinking aloud while exploring an information system , 1999 .

[32]  R. Hirschheim.  INFORMATION SYSTEMS EPISTEMOLOGY: AN HISTORICAL PERSPECTIVE , 2000 .

[33]  Paul A. Cairns,et al.  HCI... not as it should be: inferential statistics in HCI research , 2007, BCS HCI.

[34]  Sharon McDonald,et al.  Retrospective protocols in usability testing: a comparison of Post-session RTA versus Post-task RTA reports , 2016, Behav. Inf. Technol..

[35]  Tamara Munzner,et al.  Process and Pitfalls in Writing Information Visualization Research Papers , 2008, Information Visualization.

[36]  Andreas Sonderegger,et al.  The influence of age in usability testing. , 2016, Applied ergonomics.

[37]  Pam J. Mayhew,et al.  When to ask participants to think aloud:A comparative study of concurrent and retrospective think-aloud methods , 2015 .

[38]  Robert A. Virzi,et al.  Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? , 1992 .

[39]  T. Wilson.  Strangers to Ourselves: Discovering the Adaptive Unconscious , 2002 .

[40]  S. Bošnjak.  The declaration of Helsinki: The cornerstone of research ethics , 2001 .

[41]  Elizabeth D. Murphy,et al.  Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability , 2010, CHI.

[42]  Menno D. T. de Jong,et al.  Evaluating municipal websites: A methodological comparison of three think-aloud variants , 2009, Gov. Inf. Q..

[43]  Tingting Zhao,et al.  Exploring Think-Alouds in Usability Testing: An International Survey , 2012, IEEE Transactions on Professional Communication.

[44]  Pam J. Mayhew,et al.  Investigation into the impact of the usability factor on the acceptance of mobile transactions:Empirical study in Saudi Arabia , 2015 .

[45]  Gilbert Cockton,et al.  An investigation into the use of field methods in the design and evaluation of interactive systems , 2008, BCS HCI.

[46]  Pam J. Mayhew,et al.  The Impact of Usability of Online Library Catalogues on the User Performance , 2014, 2014 International Conference on Information Science & Applications (ICISA).

[47]  Patrick A. Holleran,et al.  A methodological note on pitfalls in usability testing , 1991 .

[48]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[49]  E. Guba,et al.  Competing paradigms in qualitative research. , 1994 .

[50]  A. Smith,et al.  Research Methodology: A Step-by-step Guide for Beginners , 2012 .

[51]  Joseph S. Dumas,et al.  Moderating Usability Tests: Principles and Practices for Interacting: Principles and Practices for Interacting , 2008 .

[52]  Ann Blandford,et al.  Scoping Analytical Usability Evaluation Methods: A Case Study , 2008, Hum. Comput. Interact..

[53]  Pam J. Mayhew,et al.  Eye Tracking in Retrospective Think-Aloud Usability Testing:Is there Added Value? , 2017 .

[54]  John Karat,et al.  User-Centered Software Evaluation Methodologies , 1997 .

[55]  Jared M. Spool,et al.  Testing web sites: five users is nowhere near enough , 2001, CHI Extended Abstracts.

[56]  Kent L. Norman,et al.  Levels of automation and user participation in usability testing , 2006, Interact. Comput..

[57]  George Ghinea,et al.  Software development methods and usability: Perspectives from a survey in the software industry in Norway , 2008, Interact. Comput..

[58]  Jan Stage,et al.  New techniques for usability evaluation of mobile systems , 2004, Int. J. Hum. Comput. Stud..

[59]  Menno D. T. de Jong,et al.  Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison , 2004, Interact. Comput..

[60]  Carol M. Barnum.  Usability Testing Essentials: Ready, Set...Test! , 2010 .

[61]  N. McNamara,et al.  Defining usability: quality of use or quality of experience? , 2005, IPCC 2005. Proceedings. International Professional Communication Conference, 2005..

[62]  Peter C. Wright,et al.  A Cost-Effective Evaluation Method for Use by Designers , 1991, Int. J. Man Mach. Stud..

[63]  Gilbert Cockton,et al.  Modified contextual design as a field evaluation method , 2006, NordiCHI '06.

[64]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[65]  Kasper Hornbæk,et al.  Meta-analysis of correlations among usability measures , 2007, CHI.

[66]  Janne Jul Jensen,et al.  Social Context in Usability Evaluations: Concepts, Processes and Products , 2009 .

[67]  Zhiwei Guan,et al.  The validity of the stimulated retrospective think-aloud method as measured by eye tracking , 2006, CHI.

[68]  Torkil Clemmensen,et al.  Community knowledge in an emerging online professional community : the interest in theory among Danish usability professionals , 2002 .

[69]  Marion Hansen.  Ten steps to usability testing , 1991, SIGDOC '91.

[70]  Terence Andre,et al.  Usefulness of Nonverbal Cues from Participants in Usability Testing Sessions , 2005 .

[71]  Aaron Marcus,et al.  User Interface Design's Return on Investment: Examples and Statistics , 2005 .

[72]  Morten Hertzum,et al.  Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? , 2009, Behav. Inf. Technol..

[73]  Hugh Coolican,et al.  Introduction to Research Methods in Psychology , 2006 .

[74]  Jonathan W Schooler,et al.  Introspecting in the spirit of William James: comment on Fox, Ericsson, and Best (2011). , 2011, Psychological bulletin.

[75]  Catherine Courage,et al.  Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and Techniques , 2005 .

[76]  Jacob O. Wobbrock,et al.  Understanding usability practices in complex domains , 2010, CHI.

[77]  David W. Biers,et al.  Retrospective versus Concurrent Thinking-Out-Loud in Usability Testing , 1993 .

[78]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[79]  B. Cesnik,et al.  Digital Libraries , 2001, Yearbook of Medical Informatics.

[80]  Ebba Þóra Hvannberg,et al.  Basic concepts , 2020, Fluid Flow in Porous Media.

[81]  Gitte Lindgaard,et al.  Usability testing: what have we overlooked? , 2007, CHI.

[82]  Bonnie Kaplan,et al.  Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study , 1988, MIS Q..

[83]  Candace Séguinot,et al.  Some Thoughts About Think-Aloud Protocols , 1996 .

[84]  Malcolm P. Atkinson,et al.  Comparison of evaluation methods using structured usability problem reports , 1997, Behav. Inf. Technol..

[85]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[86]  Tim May,et al.  Knowing the Social World , 1998 .

[87]  Kasper Hornbæk,et al.  Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes , 2011, Int. J. Hum. Comput. Interact..

[88]  Janice Redish,et al.  User and task analysis for interface design , 1998 .

[89]  Pam J. Mayhew,et al.  To Intervene or Not to Intervene:An Investigation of Three Think-Aloud Protocols in Usability Testing , 2017 .

[90]  Joseph A. Maxwell,et al.  Qualitative Research Design: An Interactive Approach , 1996 .