How many participants are really enough for usability studies?

The growth of the Internet and related technologies has enabled the development of a new breed of dynamic websites, applications and software products that are growing rapidly in use and that have had a great impact on many businesses. These technologies need to be continuously evaluated by usability evaluation methods (UEMs) to measure their efficiency and effectiveness, to assess user satisfaction, and ultimately to improve their quality. However, estimating the sample sizes for these methods has become the source of considerable debate at usability conferences. This paper aims to determine an appropriate sample size through empirical studies on the social network and educational domains by employing three types of UEM; it also examines further the impact of sample size on the findings of usability tests. Moreover, this paper quantifies the sample size required for the Domain Specific-to-context Inspection (DSI) method, which itself is developed through an adaptive framework. The results show that there is no certain number of participants for finding all usability problems; however, the rule of 16 4 users gains much validity in user testing. The magic number of five evaluators fails to find 80% of problems in heuristic evaluation, whereas three evaluators are enough to find 91% of usability problems in the DSI method.

[1]  Gitte Lindgaard,et al.  Usability testing: what have we overlooked? , 2007, CHI.

[2]  Peter Kulchyski.  and , 2015 .

[3]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[4]  Harry Hochheiser,et al.  Research Methods for Human-Computer Interaction , 2008 .

[5]  Dennis R. Wixon.  Evaluating usability methods: why the current literature fails the practitioner , 2003, INTR.

[6]  Gavriel Salvendy,et al.  Number of people required for usability evaluation , 2010, Commun. ACM.

[7]  Ritch Macefield,et al.  How to specify the participant group size for usability studies: a practitioner's guide , 2009 .

[8]  Robert A. Virzi,et al.  Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? , 1992 .

[9]  Martin Schmettow,et al.  Sample size in usability studies , 2012, Commun. ACM.

[10]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[11]  Gilbert Cockton,et al.  Why and when five test users aren’t enough , 2001 .

[12]  Jakob Nielsen,et al.  Determining Usability Test Sample Size , 2006 .

[13]  L. Faulkner.  Beyond the five-user assumption: Benefits of increased sample sizes in usability testing , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[14]  Saudi Arabia,et al.  The Impact of the Combination between Task Designs and Think-Aloud Approaches on Website Evaluation , 2013 .

[15]  Pam J. Mayhew,et al.  Generating a Domain Specific Inspection Evaluation Method through an Adaptive Framework , 2013 .

[16]  Pam J. Mayhew,et al.  A framework for generating a domain specific inspection evaluation method: A comparative study on social networking websites , 2013, 2013 Science and Information Conference.

[17]  Ebba Þóra Hvannberg,et al.  Analysis of combinatorial user effect in international usability tests , 2004, CHI '04.

[18]  Jared M. Spool,et al.  Testing web sites: five users is nowhere near enough , 2001, CHI Extended Abstracts.

[19]  James R. Lewis,et al.  Sample sizes for usability tests: mostly math, not magic , 2006, INTR.

[20]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[21]  M. Furlong,et al.  Eight Was Not Enough , 2009 .

[22]  Sattar J. Aboud,et al.  An Adjustable Sample Size Estimation Model for Usability Assessment , 2007 .

[23]  Rolf Molich,et al.  A critique of how to specify the participant group size for usability studies: a practitioner's guide by Macefield , 2010 .

[24]  Pam J. Mayhew,et al.  Generating an Educational Domain Checklist through an Adaptive Framework for Evaluating Educational Systems , 2013 .

[25]  Pam J. Mayhew,et al.  Generating a Domain Specific Checklist through an Adaptive Framework for Evaluating Social Networking Websites , 2013 .

[26]  Pam J. Mayhew,et al.  The Impact of Usability of Online Library Catalogues on the User Performance , 2014, 2014 International Conference on Information Science & Applications (ICISA).