User testing in industry: A case study of laboratory, workshop, and field tests

Applied user testing involves more usability evaluation methods than laboratory tests and is critically dependent upon a number of issues seldom treated in the literature. The development of the system described in this longitudinal, diary-based study evolved around five user tests: a laboratory test, a workshop test, and three field tests. The user tests had a substantial impact on the focus of the entire development effort in that 25% of the primary developer’s time was spent solving problems encountered during the tests. The laboratory test made use of set tasks and was biased toward how tasks were performed with the system, at the expense of what tasks could be performed. The workshop test was more informal and apparently led the users to adopt a more exploratory attitude. Careful arousal and management of the users’ commitment to participate actively proved essential to effective user testing, especially during the field tests.

[1]  Patricia Brooks,et al.  Adding value to usability testing , 1994 .

[2]  J R Lewis,et al.  Sample Sizes for Usability Studies: Additional Considerations , 1994, Human factors.

[3]  Jens Rasmussen,et al.  Effectiveness testing of complex systems , 1997 .

[4]  David W. Biers,et al.  Team Usability Testing: Are two Heads Better than One? , 1992 .

[5]  John C. Thomas,et al.  Minimizing ecological gaps in interface design , 1989, IEEE Software.

[6]  David W. Biers,et al.  Software Usability Testing: Do Evaluator Intervention and Task Structure Make any Difference? , 1992 .

[7]  Alan E. Benson,et al.  Are We Overlooking Some Usability Testing Methods? A Comparison of Lab, Beta, and Forum Tests , 1993, Behav. Inf. Technol..

[8]  Morten Hertzum,et al.  The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgments , 1998 .

[9]  Dieter Zirkler,et al.  Usability testing in a competive market: lessons learned , 1994, Behav. Inf. Technol..

[10]  DesurvireHeather,et al.  Usability testing vs. heuristic evaluation , 1992 .

[11]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[12]  Kuldeep Kumar,et al.  Post implementation evaluation of computer-based information systems: current practices , 1990, Commun. ACM.

[13]  Dennis R. Wixon,et al.  Making a difference—the impact of inspections , 1996, CHI.

[14]  John L. Bennett,et al.  Usability Engineering: Our Experience and Evolution , 1988 .

[15]  John Rieman,et al.  The diary study: a workplace-oriented research tool to guide laboratory efforts , 1993, INTERCHI.

[16]  Robert W. Bailey,et al.  Usability Testing vs. Heuristic Evaluation: A Head-to-Head Comparison , 1992 .

[17]  Martha R. Szczur,et al.  Usability testing - on a budget: a NASA usability test case study , 1994, Behav. Inf. Technol..

[18]  Patrick A. Holleran,et al.  A methodological note on pitfalls in usability testing , 1991 .

[19]  Jaclyn R. Schrier.  Reducing Stress Associated with Participating in a Usability Test , 1992 .

[20]  Peter C. Wright,et al.  A Cost-Effective Evaluation Method for Use by Designers , 1991, Int. J. Man Mach. Stud..

[21]  Arnold M. Lund.  Another approach to justifying the cost of usability , 1997, INTR.

[22]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.