Re-assessing the usability metric for user experience (UMUX) scale

Usability Metric for User Experience (UMUX) and its shorter form variant UMUX-LITE are recent additions to standardized usability questionnaires. UMUX aims to measure perceived usability by employing fewer items that are in closer conformance with the ISO 9241 definition of usability, while UMUX-LITE conforms to the technology acceptance model (TAM). UMUX has been criticized regarding its reliability, validity, and sensitivity, but these criticisms are mostly based on reported findings associated with the data collected by the developer of the questionnaire. Our study re-evaluates the UMUX and UMUX-LITE scales using psychometric methods with data sets acquired through two usability evaluation studies: an online word processor evaluation survey (n = 405) and a web-based mind map software evaluation survey for three applications (n = 151). Data sets yielded similar results for indicators of reliability. Both UMUX and UMUX-LITE items were sensitive to the software when the scores for the evaluated software were not very close, but we could not detect a significant difference between the software when the scores were closer. UMUX and UMUX-LITE items were also sensitive to users' level of experience with the software evaluated in this study. Neither of the scales was found to be sensitive to the participants' age, gender, or whether they were native English speakers. The scales significantly correlated with the System Usability Scale (SUS) and the Computer System Usability Questionnaire (CSUQ), indicating their concurrent validity. The parallel analysis of principal components of UMUX pointed out a single latent variable, which was confirmed through a factor analysis, that showed the data fits better to a single-dimension factor structure.

[1]  Kraig Finstad.  The system usability scale and non-native English speakers , 2006 .

[2]  Andreas M. Brandmaier,et al.  Structural Equation Modeling With Ωnyx , 2015 .

[3]  James R. Lewis,et al.  UMUX-LITE: when there's no time for the SUS , 2013, CHI.

[4]  Jum C. Nunnally,et al.  Psychometric Theory— 25 Years Ago and Now1 , 1975 .

[5]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[6]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[7]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[8]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[9]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[10]  James R. Lewis,et al.  Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies , 2002, Int. J. Hum. Comput. Interact..

[11]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[12]  Robert F. DeVellis,et al.  Scale Development: Theory and Applications. , 1992 .

[13]  Martin J. Eppler.  A Comparison between Concept Maps, Mind Maps, Conceptual Diagrams, and Visual Metaphors as Complementary Tools for Knowledge Construction and Sharing , 2006, Inf. Vis..

[14]  Stefano Federici,et al.  On the dimensionality of the System Usability Scale: a test of alternative measurement models , 2009, Cognitive Processing.

[15]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..

[16]  James R. Lewis,et al.  Measuring Perceived Usability: The SUS, UMUX-LITE, and AltUsability , 2015, Int. J. Hum. Comput. Interact..

[17]  Paul A. Cairns,et al.  A Commentary on Short Questionnaires for Assessing Usability , 2013, Interact. Comput..

[18]  Jurek Kirakowski,et al.  The Software Usability Measurement Inventory: Background and Usage , 1996 .

[19]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[20]  John J. Bosley.  Creating a Short Usability Metric for User Experience (UMUX) Scale , 2013, Interact. Comput..

[21]  Stefano Federici,et al.  Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience , 2015, Int. J. Hum. Comput. Interact..

[22]  Kraig Finstad,et al.  Response to commentaries on 'The Usability Metric for User Experience' , 2013, Interact. Comput..

[23]  James R. Lewis.  Critical Review of 'The Usability Metric for User Experience' , 2013, Interact. Comput..

[24]  Thomas K. Landauer,et al.  Research Methods in Human-Computer Interaction , 1988 .

[25]  Jeff Sauro,et al.  Standardized Usability Questionnaires , 2012 .

[26]  James B. Schreiber,et al.  Reporting Structural Equation Modeling and Confirmatory Factor Analysis Results: A Review , 2006 .