Measuring Perceived Usability: The SUS, UMUX-LITE, and AltUsability

The purpose of this research was to investigate various measurements of perceived usability, in particular, to assess (a) whether a regression formula developed previously to bring Usability Metric for User Experience LITE (UMUX-LITE) scores into correspondence with System Usability Scale (SUS) scores would continue to do so accurately with an independent set of data; (b) whether additional items covering concepts such as findability, reliability, responsiveness, perceived use by others, effectiveness, and visual appeal would be redundant with the construct of perceived usability or would align with other potential constructs; and (c) the dimensionality of the SUS as a function of self-reported frequency of use and expertise. Given the broad use of and emerging interpretative norms for the SUS, it was encouraging that the regression equation for the UMUX-LITE worked well with this independent set of data, although there is still a need to investigate its efficacy with a broader set of products and methods. Results from a series of principal components analyses indicated that most of the additional concepts, such as findability, familiarity, efficiency, control, and visual appeal covered the same statistical ground as the other more standard metrics for perceived usability. Two of the other items (Reliable and Responsive) made up a reliable construct named System Quality. None of the structural analyses of the SUS as a function of frequency of use or self-reported expertise produced the expected components, indicating the need for additional research in this area and a need to be cautious when using the Usable and Learnable components described in previous research.

[1]  Jeff Sauro,et al.  Quantifying the User Experience: Practical Statistics for User Research , 2012 .

[2]  James R. Lewis.  Tradeoffs in the Design of the IBM Computer Usability Satisfaction Questionnaires , 1999, HCI.

[3]  Fred Reichheld,et al.  The Ultimate Question: Driving Good Profits and True Growth , 2006 .

[4]  Stefano Federici,et al.  On the dimensionality of the System Usability Scale: a test of alternative measurement models , 2009, Cognitive Processing.

[5]  N. Schmitt.  Uses and abuses of coefficient alpha. , 1996 .

[6]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[7]  J. Nunnally,et al.  Psychometric Theory: NY. , 1978 .

[8]  Jeff Sauro,et al.  Correlations among prototypical usability metrics: evidence for the construct of usability , 2009, CHI.

[9]  Kraig Finstad.  The system usability scale and non-native English speakers , 2006 .

[10]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[11]  Thomas S. Tullis,et al.  A Comparison of Questionnaires for Assessing Website Usability , 2004 .

[12]  Rebecca A. Grier,et al.  The System Usability Scale , 2013 .

[13]  James R. Lewis,et al.  Usability: Lessons Learned … and Yet to Be Learned , 2014, Int. J. Hum. Comput. Interact..

[14]  James R. Lewis,et al.  UMUX-LITE: when there's no time for the SUS , 2013, CHI.

[15]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[16]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[17]  F. Reichheld.  The one number you need to grow. , 2003, Harvard business review.

[18]  Joseph B. Sidowski,et al.  Measurements of computer satisfaction, literacy, and aptitudes: A review , 1990, Int. J. Hum. Comput. Interact..

[19]  Stefano Federici,et al.  Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience , 2015, Int. J. Hum. Comput. Interact..

[20]  Kraig Finstad,et al.  Response to commentaries on 'The Usability Metric for User Experience' , 2013, Interact. Comput..

[21]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[22]  Jeff Sauro,et al.  When designing usability questionnaires, does it hurt to be positive? , 2011, CHI.

[23]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[24]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[25]  James R. Lewis,et al.  Multipoint scales: Mean and median differences and observed significance levels , 1993, Int. J. Hum. Comput. Interact..

[26]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[27]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[28]  James R. Lewis.  Critical Review of 'The Usability Metric for User Experience' , 2013, Interact. Comput..

[29]  Sarah Diefenbach,et al.  The 'hedonic' in human-computer interaction: history, contributions, and future research directions , 2014, Conference on Designing Interactive Systems.

[30]  Jen-Her Wu,et al.  Empirical evaluation of the revised end user computing acceptance model , 2004, Comput. Hum. Behav..

[31]  James R. Lewis,et al.  Investigating the Correspondence Between UMUX-LITE and SUS Scores , 2015, HCI.

[32]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..