It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field

The distinction between field and laboratory is classical in research methodology. In human-computer interaction, and in usability evaluation in particular, it has been a controversial topic for several years. The advent of mobile devices has revived this topic. Empirical studies that compare evaluations in the two settings are beginning to appear, but they provide very different results. This paper presents results from an experimental comparison of a field-based and a lab-based usability evaluation of a mobile system. The two evaluations were conducted in exactly the same way. The conclusion is that it is definitely worth the hassle to conduct usability evaluations in the field. In the field-based evaluation we identified significantly more usability problems and this setting revealed problems with interaction style and cognitive load that were not identified in the laboratory.

[1]  Jan Stage,et al.  New techniques for usability evaluation of mobile systems , 2004, Int. J. Hum. Comput. Stud..

[2]  Mik Lamming,et al.  Interactive system design , 1995 .

[3]  M. H. Heycock,et al.  Papers , 1971, BMJ : British Medical Journal.

[4]  Stephen A. Brewster,et al.  Gestural and audio metaphors as a means of control for mobile devices , 2002, CHI.

[5]  Morten Hertzum,et al.  User testing in industry: A case study of laboratory, workshop, and field tests , 1999 .

[6]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[7]  Anu Kankainen,et al.  Usability testing of mobile applications: a comparison between laboratory and field testing , 2005 .

[8]  Andreas Butz,et al.  Location-Aware Shopping Assistance: Evaluation of a Decision-Theoretic Approach , 2002, Mobile HCI.

[9]  Jesper Kjeldskov,et al.  Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field , 2004, Mobile HCI.

[10]  P. Hancock,et al.  Human Mental Workload , 1988 .

[11]  Gregory D. Abowd,et al.  Charting past, present, and future research in ubiquitous computing , 2000, TCHI.

[12]  Michael Sannella,et al.  Constraint satisfaction and debugging for interactive user interfaces , 1994 .

[13]  Klaus Kaasgaard,et al.  Comparative usability evaluation , 2004, Behav. Inf. Technol..

[14]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[15]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[16]  Oskar Juhlin,et al.  Motorcyclists Using Hocman - Field Trials on Mobile Interaction , 2003, Mobile HCI.

[17]  Jesper Kjeldskov,et al.  Creating Realistic Laboratory Settings: Comparative Studies of Three Think-Aloud Usability Evaluations of a Mobile System , 2003, INTERACT.

[18]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[19]  Dario D. Salvucci.  Predicting the effects of in-car interfaces on driver behavior using a cognitive architecture , 2001, CHI.

[20]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[21]  Connor Graham,et al.  Mobile evaluation: what the data and the metadata told us , 2003 .

[22]  Karen Cheng,et al.  On the road and on the Web?: comprehension of synthetic and human speech while driving , 2001, CHI.

[23]  Clare-Marie Karat,et al.  Comparison of empirical testing and walkthrough methods in user interface evaluation , 1992, CHI.

[24]  Lynne Baillie.  Future Telecommunications: Exploring Actual Use , 2003, INTERACT.

[25]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[26]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[27]  Stephen A. Brewster,et al.  Overcoming the Lack of Screen Space on Mobile Computers , 2002, Personal and Ubiquitous Computing.