Measuring user experience in automated driving: Developing a single-item measure

Measuring user experience is highly important for human-centered development and thus for designing automated driving systems. Multi-item measures such as the System Usability Scale (SUS) [7] or the Usability Metric for User Experience (UMUX) [14] are commonly used for collecting user feedback on technical systems or products. The goal of the present study was to investigate the potentials of a single-item approach as an economic alternative for measuring user experience compared to multi-item scales. Therefore, a single-item measure was developed to assess both event-related and cumulative user experience in automated driving. User experience was manipulated in a between-subject design implemented in a real-world driving task and feedback was collected using the newly developed Single Item User Experience (SIUX) scale, the UMUX, and the SUS. Results indicate that the SIUX scale is more sensitive than the UMUX to differences in event-related user experience, but not in cumulative user experience. Both the SIUX and the UMUX were more sensitive than the SUS when measuring differences in cumulative user experience. Future studies should be aimed at investigating the applicability of the SIUX scale to domains other than automated driving and at collecting more extensive data on validity and reliability of all three instruments.

[1]  G. Glass,et al.  Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance , 1972 .

[2]  M K Kaiser,et al.  MANOVA method for analyzing repeated measures designs: an extensive primer. , 1985, Psychological bulletin.

[3]  Susumu Saito,et al.  Ergonomics of human-system interaction , 1994 .

[4]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[5]  Veronika Weinbeer,et al.  Using the wizard of Oz paradigm to prototype automated vehicles: methodological challenges , 2019, AutomotiveUI.

[6]  B. Shamir,et al.  A single-item graphic scale for the measurement of organizational identification , 2004 .

[7]  Der Einsatz der ,Experience Sampling Method‘ in der Medienwissenschaft , 2001 .

[8]  D. Dillman,et al.  EFFECTS OF QUESTIONNAIRE LENGTH, RESPONDENT-FRIENDLY DESIGN, AND A DIFFICULT QUESTION ON RESPONSE RATES FOR OCCUPANT-ADDRESSED CENSUS MAIL SURVEYS , 1993 .

[9]  M. Nagy.  Using a single-item approach to measure facet job satisfaction , 2002 .

[10]  Patricia S. Jones,et al.  An Adaptation of Brislin’s Translation Model for Cross-cultural Research , 2001, Nursing research.

[11]  Mohammad Zarour,et al.  User experience framework that combines aspects, dimensions, and measurement methods , 2017 .

[12]  Rex Hartson,et al.  The UX book, process and guidelines for ensuring a quality user experience by Rex Hartson and Pardha S. Pyla , 2012, SOEN.

[13]  Daniel Kahneman,et al.  Living, and thinking about it: two perspectives on life , 2005 .

[14]  D. G. Morrison,et al.  Do We Really Need Multiple-Item Measures in Service Research? , 2001 .

[15]  J. P. Wanous,et al.  Overall job satisfaction: how good are single-item measures? , 1997, The Journal of applied psychology.

[16]  Nigel Bevan,et al.  Classifying and selecting UX and usability measures , 2008 .

[17]  G. Box.  NON-NORMALITY AND TESTS ON VARIANCES , 1953 .

[18]  Evangelos Karapanos,et al.  User experience over time , 2008, CHI Extended Abstracts.

[19]  Klaus Bengler,et al.  Measurement of momentary user experience in an automotive context , 2013, AutomotiveUI.

[20]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[21]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[22]  Dilek Karahoca,et al.  Re-assessing the usability metric for user experience (UMUX) scale , 2016 .

[23]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[24]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[25]  Alexandra Neukum,et al.  Driver compliance to take-over requests with different auditory outputs in conditional automation. , 2017, Accident; analysis and prevention.

[26]  James R. Lewis,et al.  Investigating the Correspondence Between UMUX-LITE and SUS Scores , 2015, HCI.

[27]  Thomas P. Moran,et al.  New Perspectives on Human-Computer Interaction. , 1985 .

[28]  John P. Campbell,et al.  JOB SATISFACTION: ARE ALL THE PARTS THERE? , 1983 .

[29]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..

[30]  H. Keselman,et al.  Is the ANOVA F-Test Robust to Variance Heterogeneity When Sample Sizes are Equal?: An Investigation via a Coefficient of Variation , 1977 .

[31]  Klaus Bengler,et al.  The Renaissance of Wizard of Oz (WoOz) – Using the WoOz methodology to prototype automated vehicles , .

[32]  Adamantios Diamantopoulos,et al.  Using single-item measures for construct measurement in management research Conceptual issues and application guidelines , 2009 .

[33]  J. P. Wanous,et al.  Single-Item Reliability: A Replication and Extension , 2001 .

[34]  Stefano Federici,et al.  Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience , 2015, Int. J. Hum. Comput. Interact..

[35]  Eli P. Cox,et al.  The Optimal Number of Response Alternatives for a Scale: A Review , 1980 .

[36]  James R. Lewis,et al.  The System Usability Scale: Past, Present, and Future , 2018, Int. J. Hum. Comput. Interact..