Are We Testing Utility? Analysis of Usability Problem Types

Usability problems and related redesign recommendations are the main outcome of usability tests although both are questioned in terms of impact in the design process. Problem classifications aim to provide better feedback for designers by improving usability problem identification, analysis and reporting. However, within the classifications, quite little is discussed about the types and the contents of usability problems as well as the types of required design efforts. We address this problem by scrutinizing the findings of three empirical usability tests conducted in software development projects. As a result, 173 problems were classified into 11 categories. Specific focus was placed on the distinction between the utility and usability types of problems, in order to define the correct development phase and method to fix the problem. The number of utility problems varied from 51 % to 74 %, which shows that early usability testing with a think-aloud protocol and an open task structure measure both utility and usability equally well.

[1]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1992 .

[2]  Kasper Hornbæk,et al.  Analysis in usability evaluations: an exploratory study , 2010, NordiCHI.

[3]  Kasper Hornbæk,et al.  Dogmas in the assessment of usability evaluation methods , 2010, Behav. Inf. Technol..

[4]  Malcolm P. Atkinson,et al.  Comparison of evaluation methods using structured usability problem reports , 1997, Behav. Inf. Technol..

[5]  D. Campbell.  Task Complexity: A Review and Analysis , 1988 .

[6]  John L. Bennett,et al.  Usability Engineering: Our Experience and Evolution , 1988 .

[7]  Steven M. Belz,et al.  The user action framework: a reliable foundation for usability engineering support tools , 2001, Int. J. Hum. Comput. Stud..

[8]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.

[9]  Klaus Kaasgaard,et al.  Comparative usability evaluation , 2004, Behav. Inf. Technol..

[10]  Nigel Bevan,et al.  Usability is Quality of Use , 1995 .

[11]  Dennis R. Wixon.  Evaluating usability methods: why the current literature fails the practitioner , 2003, INTR.

[12]  Richard Baskerville,et al.  Investigating Information Systems with Action Research , 1999, Commun. Assoc. Inf. Syst..

[13]  Ebba Þóra Hvannberg,et al.  Classification of Usability Problems (CUP) Scheme , 2003, INTERACT.

[14]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[15]  Mo Adam Mahmood,et al.  Variables affecting information technology end-user satisfaction: a meta-analysis of the empirical literature , 2000, Int. J. Hum. Comput. Stud..

[16]  Jonathan Grudin,et al.  Utility and Usability: Research Issues and Development Contexts , 1992, Interact. Comput..

[17]  Dennis G. Kafura,et al.  The Usability Problem Taxonomy: A Framework for Classification and Analysis , 1999, Empirical Software Engineering.

[18]  Kasper Hornbæk,et al.  Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes , 2011, Int. J. Hum. Comput. Interact..

[19]  Ebba Þóra Hvannberg,et al.  Classification of usability problems (CUP) scheme: augmentation and exploitation , 2006, NordiCHI '06.

[20]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[21]  Kasper Hornbæk,et al.  Must evaluation methods be about usability? Devising and assessing the utility inspection method , 2014, Behav. Inf. Technol..

[22]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[23]  Nancy C. Goodwin,et al.  Functionality and usability , 1987, CACM.

[24]  Morten Hertzum,et al.  What you get is what you see: revisiting the evaluator effect in usability tests , 2014, Behav. Inf. Technol..

[25]  Effie Lai-Chong Law.  Evaluating the Downstream Utility of User Tests and Examining the Developer Effect: A Case Study , 2006, Int. J. Hum. Comput. Interact..

[26]  Robert B. Grady,et al.  Software Failure Analysis for High-Return Process Improvement Decisions , 1996 .

[27]  Jeff Tian,et al.  In-process Usability Problem Classification, Analysis and Improvement , 2014, 2014 14th International Conference on Quality Software.

[28]  Kasper Hornbæk,et al.  Comparing usability problems and redesign proposals as input to practical systems development , 2005, CHI.

[29]  Ann Blandford,et al.  Evaluating system utility and conceptual fit using CASSM , 2008, Int. J. Hum. Comput. Stud..

[30]  Kimmo Tarkkanen,et al.  Back to User-Centered Usability Testing , 2013, SouthCHI.

[31]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[32]  Ebba Þóra Hvannberg,et al.  Assessing the reliability, validity and acceptance of a classification scheme of usability problems (CUP) , 2014, J. Syst. Softw..

[33]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[34]  Dong-Han Ham.  A model-based framework for classifying and diagnosing usability problems , 2013, Cognition, Technology & Work.

[35]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1993 .