Must evaluation methods be about usability? Devising and assessing the utility inspection method

Whereas research in usability evaluation abounds, few evaluation approaches focus on utility. We present the utility inspection method (UIM), which prompts evaluators about the utility of the system they evaluate. The UIM asks whether a system uses global platforms, provides support infrastructure, is robust, gives access to rich content, allows customisation, offers symbolic value and supports companionship among users and between users and developers. We compare 47 participants’ use of UIM and heuristic evaluation (HE). The UIM helps identify more than three times as many problems as HE about the context of activities; HE helps identify 2.5 times as many problems as UIM about the interface. Usability experts consider the problems found with UIM more severe and more complex to solve compared to those found with HE. We argue that UIM complements existing usability evaluation methods and discuss future research on utility inspection.

[1]  Gilbert Cockton,et al.  Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation , 2001, BCS HCI/IHM.

[2]  Melissa A. Federoff,et al.  HEURISTICS AND USABILITY GUIDELINES FOR THE CREATION AND EVALUATION OF FUN IN VIDEO GAMES , 2002 .

[3]  Martin R. Gibbs,et al.  Mediating intimacy: designing technologies to support strong-tie relationships , 2005, CHI.

[4]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[5]  Ken Eason,et al.  Towards the experimental study of usability , 1984 .

[6]  Dominique L. Scapin,et al.  Ergonomic criteria for evaluating the ergonomic quality of interactive systems , 1997, Behav. Inf. Technol..

[7]  Lars Mathiassen,et al.  Using Computers in Qualitative Research , 1991 .

[8]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[9]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[10]  Ebba Þóra Hvannberg,et al.  A Green Paper on Usability Maturation , 2008, Maturing Usability.

[11]  Anind K. Dey,et al.  Heuristic evaluation of ambient displays , 2003, CHI '03.

[12]  Jonathan Grudin,et al.  Utility and Usability: Research Issues and Development Contexts , 1992, Interact. Comput..

[13]  Joseph S. Dumas,et al.  User-based evaluations , 2002 .

[14]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[15]  William Buxton,et al.  Usability evaluation considered harmful (some of the time) , 2008, CHI.

[16]  Kasper Hornbæk,et al.  Comparing usability problems and redesign proposals as input to practical systems development , 2005, CHI.

[17]  Jean Scholtz,et al.  Usability Evaluation , 2001 .

[18]  Clayton Lewis,et al.  Designing for usability—key principles and what designers think , 1983, CHI '83.

[19]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[20]  N. Hoffart.  Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[21]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[22]  Kasper Hornbæk,et al.  Making use of business goals in usability evaluation: an experiment with novice evaluators , 2008, CHI.

[23]  Wendy E. Mackay,et al.  Generative walkthroughs: to support creative redesign , 2009, C&C '09.

[24]  Jonathan Grudin,et al.  Organizational Issues in Development and Implementation of Interactive Systems , 1997 .

[25]  Avi Parush,et al.  Utility and Experience in the Evolution of Usability , 2008, Maturing Usability.

[26]  Ann Blandford,et al.  Evaluating system utility and conceptual fit using CASSM , 2008, Int. J. Hum. Comput. Stud..

[27]  Fred D. Davis,et al.  Toward preprototype user acceptance testing of new information systems: implications for software project management , 2004, IEEE Transactions on Engineering Management.

[28]  Brian Shackel,et al.  Usability - Context, framework, definition, design and evaluation , 1991, Interact. Comput..

[29]  Alistair G. Sutcliffe,et al.  Experience with user-centred requirements engineering , 2011, Requirements Engineering.

[30]  David Pinelle,et al.  Heuristic evaluation for games: usability principles for video game design , 2008, CHI.

[31]  Marc Hassenzahl,et al.  User experience - a research agenda , 2006, Behav. Inf. Technol..

[32]  John Zimmerman,et al.  User experience over time: an initial framework , 2009, CHI.

[33]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.

[34]  Kasper Hornbæk,et al.  Metaphors of human thinking for usability inspection and design , 2008, TCHI.

[35]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[36]  Kaisa Väänänen,et al.  Identifying drivers and hindrances of social user experience in web services , 2010, CHI.

[37]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .