Extra-laboratorial usability tests: An empirical comparison of remote and classical field testing with lab testing.

The present article examined the effects of using different extra-laboratorial testing procedures in usability testing. Three experiments were conducted using different artefacts (website, computer-simulated mobile phone, fully operational smartphone) to compare different methodological approaches in field testing (synchronous and asynchronous remote testing, classical field testing) to lab-based testing under different operational conditions (dual task demands, poor product usability). Typical outcome variables of usability testing were measured, including task completion time, click rate, perceived usability and workload. Overall, the results showed no differences between field and lab-based testing under favourable operational conditions. However, under difficult operational conditions (i.e. dual task demands, poor product usability) differences between field and lab-based testing emerged (corresponding to small and medium effect sizes). The findings showed a complex pattern of effects, suggesting that there was no general advantage of one testing procedure over another.

[1]  Andreas Sonderegger,et al.  The influence of product aesthetics and usability over the course of time: a longitudinal field experiment , 2012, Ergonomics.

[2]  Henry Been-Lirn Duh,et al.  Usability evaluation for mobile device: a comparison of laboratory and field tests , 2006, Mobile HCI.

[3]  Kapil Chalil Madathil,et al.  Synchronous remote usability testing: a new approach facilitated by virtual worlds , 2011, CHI.

[4]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[5]  Chris Andrzejczak,et al.  The effect of testing location on usability testing performance, participant stress levels, and subjective testing experience , 2010, J. Syst. Softw..

[6]  Kai-Christoph Hamborg,et al.  The Interplay between Usability and Aesthetics: More Evidence for the "What Is Usable Is Beautiful" Notion , 2014, Adv. Hum. Comput. Interact..

[7]  G. Matthews.  Human Performance: Cognition, Stress and Individual Differences , 2000 .

[8]  Andreas Sonderegger,et al.  The influence of laboratory set-up in usability tests: effects on user performance, subjective ratings and physiological measures , 2009, Ergonomics.

[9]  Gerrit Hirschfeld,et al.  Expected usability is not a valid indicator of experienced usability , 2015, PeerJ Comput. Sci..

[10]  James J. Lindsay,et al.  Research in the Psychological Laboratory , 1999 .

[11]  H. Heuer.  Dual-task performance. , 1996 .

[12]  Gilbert Cockton,et al.  From quality in use to value in the world , 2004, CHI EA '04.

[13]  Sangwon Lee,et al.  Understanding user preferences based on usability and aesthetics before and after actual use , 2010, Interact. Comput..

[14]  Miles MacLeod,et al.  Usability measurement in context , 1994, Behav. Inf. Technol..

[15]  R. L. Dipboye,et al.  Research settings in industrial and organizational psychology: Are findings in the field more generalizable than in the laboratory? , 1979 .

[16]  Erich Kirchler,et al.  Price and its Relation to Objective and Subjective Product Quality: Evidence from the Austrian Market , 2010 .

[17]  Steve Howard,et al.  Evaluating the usability of a mobile guide: The influence of location, participants and resources , 2005, Behav. Inf. Technol..

[18]  Jan Stage,et al.  It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field , 2006, NordiCHI '06.

[19]  Daniel G Bobrow,et al.  On data-limited and resource-limited processes , 1975, Cognitive Psychology.

[20]  Jürgen Sauer,et al.  The influence of user expertise and prototype fidelity in usability tests. , 2010, Applied ergonomics.

[21]  Thomas S. Tullis,et al.  An Empirical Comparison of Lab and Remote Usability Testing of Web Sites , 2002 .

[22]  Pam J. Mayhew,et al.  A Comparative Study of Synchronous and Asynchronous Remote Usability Testing Methods , 2013 .

[23]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[24]  Andreas Sonderegger,et al.  The influence of age in usability testing. , 2016, Applied ergonomics.

[25]  Lan Xia,et al.  The Price is Unfair! A Conceptual Framework of Price Fairness Perceptions , 2004 .

[26]  Claude J. Elie,et al.  Remote Usability Evaluation: Overview and Case Studies , 2002, Int. J. Hum. Comput. Interact..

[27]  Jan Stage,et al.  New techniques for usability evaluation of mobile systems , 2004, Int. J. Hum. Comput. Stud..

[28]  Joachim Meyer,et al.  Economic and subjective measures of the perceived value of aesthetics and usability , 2006, TCHI.

[29]  H. Pashler.  Dual-task interference in simple tasks: data and theory. , 1994, Psychological bulletin.

[30]  R. Stennett,et al.  The relationship of performance level to level of arousal. , 1957, Journal of experimental psychology.

[31]  N. Tractinsky,et al.  What is beautiful is usable , 2000, Interact. Comput..

[32]  Andreas Sonderegger,et al.  The influence of socio-cultural background and product value in usability testing. , 2013, Applied ergonomics.

[33]  Alexandre N. Tuch,et al.  Is beautiful really usable? Toward understanding the relation between usability, aesthetics, and affect in HCI , 2012, Comput. Hum. Behav..

[34]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[35]  D. Dimitrov,et al.  Pretest-posttest designs and measurement of change. , 2003, Work.

[36]  Jan Stage,et al.  What happened to remote usability testing?: an empirical study of three methods , 2007, CHI.

[37]  Anu Kankainen,et al.  Usability testing of mobile applications: a comparison between laboratory and field testing , 2005 .

[38]  Andreas Sonderegger,et al.  Methodological issues in product evaluation: the influence of testing environment and task scenario. , 2011, Applied ergonomics.

[39]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..