Making use of business goals in usability evaluation: an experiment with novice evaluators

The utility and impact of a usability evaluation depend on how well its results align with the business goals of the system under evaluation. However, how to achieve such alignment is not well understood. We propose a simple technique that requires active consideration of a system's business goals in planning and reporting evaluations. The technique is tested in an experiment with 44 novice evaluators using think aloud testing. The evaluators considering business goals report fewer usability problems compared to evaluators that did not use the technique. The company commissioning the evaluation, however, assesses those problems 30-42% higher on four dimensions of utility. We discuss how the findings may generalize to usability professionals, and how the technique may be used in realistic usability evaluations. More generally, we discuss how our results illustrate one of a variety of ways in which business goals and other facets of a system's context may enter into usability evaluations.

[1]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[2]  Erik Frøkjær,et al.  Usability and Software Development: Roles of the Stakeholders , 2007, HCI.

[3]  Chauncey E. Wilson,et al.  The Usability Engineering Framework for Product Design and Evaluation , 1997 .

[4]  Lynne Hall,et al.  Changing Analysts' Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment , 2004 .

[5]  Ann Blandford,et al.  Usability Work in Professional Website Design: Insights from Practitioners' Perspectives , 2008, Maturing Usability.

[6]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.

[7]  Joseph S. Dumas,et al.  Usability in practice: formative usability evaluations - evolution and revolution , 2002, CHI Extended Abstracts.

[8]  Jesper Simonsen,et al.  Involving top management in IT projects , 2007, CACM.

[9]  Kasper Hornbæk,et al.  Two psychology-based usability inspection techniques studied in a diary experiment , 2004, NordiCHI '04.

[10]  Sidney L. Smith,et al.  Guidelines for Designing User Interface Software , 1986 .

[11]  Carl Gutwin,et al.  Groupware walkthrough: adding context to groupware usability evaluation , 2002, CHI.

[12]  Gilbert Cockton,et al.  Inspection-based evaluations , 2002 .

[13]  Gilbert Cockton,et al.  Reconditioned merchandise: extended structured report formats in usability inspection , 2004, CHI EA '04.

[14]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[15]  Jan Gulliksen,et al.  Usability professionals - current practices and future development , 2006, Interact. Comput..

[16]  Robin Jeffries,et al.  Usability problem reports: helping evaluators communicate effectively with developers , 1994 .

[17]  Bonnie E. John,et al.  Tracking the effectiveness of usability evaluation methods , 1997, Behav. Inf. Technol..

[18]  Jan Stage,et al.  The Interplay Between Usability Evaluation and User Interaction Design , 2006, Int. J. Hum. Comput. Interact..

[19]  AsbjØRn F⊘lstad,et al.  Work-Domain Experts as Evaluators: Usability Inspection of Domain-Specific Work-Support Systems , 2007 .

[20]  Bonnie E. John,et al.  Learning and using the cognitive walkthrough method: a case study approach , 1995, CHI '95.

[21]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2003, Int. J. Hum. Comput. Interact..

[22]  Joseph S. Dumas,et al.  Oracle Corporation , 1994 .

[23]  Kasper Hornbæk,et al.  Comparing usability problems and redesign proposals as input to practical systems development , 2005, CHI.

[24]  Joseph W. Janes,et al.  Relevance judgments and the incremental presentation of document representations , 1991, Inf. Process. Manag..

[25]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[26]  John L. Bennett,et al.  Usability Engineering: Our Experience and Evolution , 1988 .

[27]  Joseph S. Dumas,et al.  Making usability recommendations useful and usable , 2007 .

[28]  Jesper Simonsen,et al.  Participatory IT Design: Designing for Business and Workplace Realities , 2004 .

[29]  Phil Carter.  Liberating usability testing , 2007, INTR.

[30]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[31]  Dana Chisnell,et al.  Handbook of Usability Testing , 2009 .

[32]  Jesper Simonsen,et al.  Participatory IT Design ‑ an exemplary case , 2004 .

[33]  Sarah A. Bloomer,et al.  Pitching usability to your organization , 1997, INTR.

[34]  Jakob Nielsen,et al.  E-Commerce User Experience , 2001 .

[35]  Cathleen Wharton,et al.  The cognitive walkthrough method: a practitioner's guide , 1994 .

[36]  Kim Halskov,et al.  Methods & tools: context: an active choice in usability work , 1998, INTR.

[37]  Heather Desurvire,et al.  EMPIRICISM VERSUS JUDGEMENT: COMPARING USER INTERFACE EVALUATION METHODS ON A NEW TELEPHONE-BASED INTERFACE , 1991, SGCH.

[38]  Alex S. Taylor,et al.  Towards a Methodology Employing Critical Parameters to Deliver Performance Improvements in Interactive Systems , 1999, INTERACT.

[39]  Rick Kazman,et al.  Categorizing Business Goals for Software Architectures , 2005 .

[40]  Gilbert Cockton,et al.  Designing worth is worth designing , 2006, NordiCHI '06.

[41]  Martin C. Maguire,et al.  Context of Use within usability activities , 2001, Int. J. Hum. Comput. Stud..

[42]  MolichRolf,et al.  Describing usability problems , 2004 .

[43]  Effie Lai-Chong Law.  Evaluating the Downstream Utility of User Tests and Examining the Developer Effect: A Case Study , 2006, Int. J. Hum. Comput. Interact..