A Systematic and Generalizable Approach to the Heuristic Evaluation of User Interfaces

ABSTRACT Heuristic evaluation is one of the most actively used techniques for analyzing usability, as it is quick and inexpensive. This technique is based on following a given set of heuristics, which are typically defined as broad rules of thumb. In this paper, we propose a systematic and generalizable approach to this type of evaluation based on using comprehensive taxonomies as a source for the heuristics. This approach contrasts with other typical approaches, such as following (or adapting) Jakob Nielsen’s heuristics or creating ad hoc heuristics (formally or informally). The usefulness of our approach is investigated in two ways. Firstly, we carry out an actual heuristic evaluation of a mobile app in this manner, which we describe in detail. Secondly, we compare our approach and Nielsen’s. Additionally, we identify some limitations in Nielsen’s heuristics and some inconsistencies between them and established usability models, including Nielsen’s own.

[1]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[2]  Stefan Wagner,et al.  A Comprehensive Model of Usability , 2007, EHCI/DS-VIS.

[3]  Hend Suliman Al-Khalifa,et al.  A heuristic checklist for usability evaluation of Saudi government mobile applications , 2016, iiWAS.

[4]  Steven Pemberton,et al.  Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification , 2010 .

[5]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[6]  Jean Vanderdonckt,et al.  A Comparative Usability Study of Electronic Newspapers , 2000, TFWWG.

[7]  Ebba Þóra Hvannberg,et al.  Heuristic evaluation: Comparing ways of finding and reporting usability problems , 2007, Interact. Comput..

[8]  Harvey P. Siy,et al.  The Chicken and the Pig: User Involvement in Developing Usability Heuristics , 2013, AMCIS.

[9]  David Raneburger,et al.  A User Study with GUIs Tailored for Smartphones , 2013, INTERACT.

[10]  Myung Hwan Yun,et al.  Evaluation of product usability: development and validation of usability dimensions and design elements based on empirical models , 2000 .

[11]  Vicente Moret-Bonillo,et al.  Usability: A Critical Analysis and a Taxonomy , 2009, Int. J. Hum. Comput. Interact..

[12]  Hermann Kaindl,et al.  A user study on tailoring GUIs for smartphones , 2014, SAC.

[13]  Frederick G. Conrad,et al.  A heuristic evaluation of a World Wide Web prototype , 1996, INTR.

[14]  Carl Gutwin,et al.  Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration , 2001, EHCI.

[15]  Vicente Moret-Bonillo,et al.  A Context-of-Use Taxonomy for Usability Studies , 2010, Int. J. Hum. Comput. Interact..

[16]  Rosa Yáñez Gómez,et al.  Heuristic Evaluation on Mobile Interfaces: A New Checklist , 2014, TheScientificWorldJournal.

[17]  Heather Desurvire,et al.  Using heuristics to evaluate the playability of games , 2004, CHI EA '04.

[18]  Vicente Moret-Bonillo,et al.  Usability evaluation and development of heuristics for second-screen applications , 2017, SAC.

[19]  Rex B. Kline,et al.  Usability measurement and metrics: A consolidated model , 2006, Software Quality Journal.

[20]  Cristian Rusu,et al.  How to develop usability heuristics: A systematic literature review , 2017, Comput. Stand. Interfaces.

[21]  Michael J. Muller,et al.  Methods & tools: participatory heuristic evaluation , 1998, INTR.

[22]  Nigel Bevan,et al.  Extending Quality in Use to Provide a Framework for Usability Measurement , 2009, HCI.

[23]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[24]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[25]  Jakob Nielsen,et al.  Guerrilla HCI: using discount usability engineering to penetrate the intimidation barrier , 1994 .

[26]  Hermann Kaindl,et al.  A User Study with GUIs Tailored for Smartphones and Tablet PCs , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[27]  André de Lima Salgado,et al.  Evolving Heuristic Evaluation for multiple contexts and audiences: Perspectives from a mapping study , 2016, SIGDOC.

[28]  Alain Abran,et al.  Usability Meanings and Interpretations in ISO Standards , 2003, Software Quality Journal.

[29]  James R. Lewis,et al.  Usability: Lessons Learned … and Yet to Be Learned , 2014, Int. J. Hum. Comput. Interact..

[30]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[31]  Virginica Rusu,et al.  A Methodology to Establish Usability Heuristics , 2011, ACHI 2011.

[32]  Kasper Hornbæk,et al.  Must evaluation methods be about usability? Devising and assessing the utility inspection method , 2014, Behav. Inf. Technol..

[33]  Stephen Kimani,et al.  Appropriating and assessing heuristics for mobile computing , 2006, AVI '06.

[34]  Ebba Þóra Hvannberg,et al.  Exploitation of heuristics for virtual environments , 2012, NordiCHI.

[35]  Vimla L. Patel,et al.  Using usability heuristics to evaluate patient safety of medical devices , 2003, J. Biomed. Informatics.

[36]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[37]  Joseph S. Dumas,et al.  The great leap forward: the birth of the usability profession (1988-1993) , 2007 .

[38]  Anind K. Dey,et al.  Heuristic evaluation of ambient displays , 2003, CHI '03.

[39]  Dalenca Pottas,et al.  A three-phase process to develop heuristics for specific application domains , 2011 .

[40]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[41]  Paula Alexandra Silva,et al.  Something Old, Something New, Something Borrowed: gathering experts' feedback while performing heuristic evaluation with a list of heuristics targeted at older adults , 2014, ACE '14 Workshops.

[42]  Michael D. Byrne,et al.  The tangled Web we wove: a taskonomy of WWW use , 1999, CHI '99.

[43]  Alistair G. Sutcliffe,et al.  Heuristic evaluation of virtual reality applications , 2004, Interact. Comput..

[44]  Dominique L. Scapin,et al.  Evaluating a user interface with ergonomic criteria , 1995, Int. J. Hum. Comput. Interact..

[45]  Melody Y. Ivory.  Web TANGO: towards automated comparison of information-centric web site designs , 2000, CHI Extended Abstracts.

[46]  David G. Novick,et al.  Usability inspection methods after 15 years of research and practice , 2007, SIGDOC '07.

[47]  Ana Isabel Veloso,et al.  Social games and different generations: A heuristic evaluation of Candy Crush Saga , 2016, 2016 1st International Conference on Technology and Innovation in Sports, Health and Wellbeing (TISHW).

[48]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[49]  G. D. Magoulas,et al.  Integrating Layered and Heuristic Evaluation for Adaptive Learning Environments , 2003 .

[50]  Vicente Moret-Bonillo,et al.  A Taxonomy-Based Usability Study of an Intelligent Speed Adaptation Device , 2014, Int. J. Hum. Comput. Interact..

[51]  Susan Harker,et al.  ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998? , 2015, HCI.

[52]  David Alonso Ríos.  Development of usability and context-of-use taxonomies, integration with techniques for the study of usability and application to real-world intelligent systems , 2015 .

[53]  Sidney L. Smith,et al.  Guidelines for Designing User Interface Software , 1986 .