Self-report measures for the assessment of human–machine interfaces in automated driving

For a successful market introduction of Level 3 Automated Driving Systems (L3 ADS), a careful evaluation of human–machine interfaces (HMIs) is necessary. User preference has often focused on usability, user experience, acceptance and trust. However, a thorough evaluation of measures when applied to ADS HMIs is missing. We investigated the appropriateness of nine self-reported measures in terms of reliability and validity. A sample of N = 57 participants completed two 15-min simulator drives with a L3 ADS. They experienced two variations of a HMI that differed in the degree of complying with common guidelines. Consistency analysis identified scales that showed insufficient reliability. Validity examination revealed a three-factorial structure of self-reports for construct validity. These factors are design-orientation, usability-orientation and acceptance-orientation. All measures were sensitive to the HMI manipulation and therefore exhibited criterion-related validity. The present study provides researchers and practitioners in the area of ADS with a recommendation for self-report measure application.

[1]  Josef F. Krems,et al.  Empirical Validation of a Checklist for Heuristic Evaluation of Automated Vehicle HMIs , 2019, AHFE.

[2]  Anne Marsden,et al.  International Organization for Standardization , 2014 .

[3]  Josef F. Krems,et al.  Learning to use automation: Behavioral changes in interaction with automated driving systems , 2019 .

[4]  Kasper Hornbæk,et al.  Meta-analysis of correlations among usability measures , 2007, CHI.

[5]  Lesley Strawderman,et al.  Assessing the utility of TAM, TPB, and UTAUT for advanced driver assistance systems. , 2017, Accident; analysis and prevention.

[6]  Johannes Kraus,et al.  Calibration of Trust Expectancies in Conditionally Automated Driving by Brand, Reliability Information and Introductionary Videos: An Online Study , 2018, AutomotiveUI.

[7]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[8]  Colin G. Drury,et al.  Foundations for an Empirically Determined Scale of Trust in Automated Systems , 2000 .

[9]  Philipp Wintersberger,et al.  Driving Hotzenplotz: A Hybrid Interface for Vehicle Control Aiming to Maximize Pleasure in Highway Driving , 2017, AutomotiveUI.

[10]  Michael Minge,et al.  The meCUE Questionnaire: A Modular Tool for Measuring User Experience , 2017 .

[11]  Austin Henderson,et al.  What you see, some of what's in the future, and how we go about doing it: HI at Apple Computer , 1995, CHI 95 Conference Companion.

[12]  Klaus Bengler,et al.  It's Out of Our Hands Now! Effects of Non-Driving Related Tasks During Highly Automated Driving on Drivers' Fatigue , 2017 .

[13]  T. Kline.  The team player inventory: Reliability and validity of a measure of predisposition toward organizational team-working environments , 1999 .

[14]  Tim Horberry,et al.  Driver Acceptance of New Technology: Theory, Measurement and Optimisation , 2017 .

[15]  M. Nees.  Acceptance of Self-driving Cars , 2016 .

[16]  Riender Happee,et al.  Public opinion on automated driving: results of an international questionnaire among 5000 respondents , 2015 .

[17]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[18]  Marc Hassenzahl,et al.  The Effect of Perceived Hedonic Quality on Product Appealingness , 2001, Int. J. Hum. Comput. Interact..

[19]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[20]  Josef F. Krems,et al.  Prior Familiarization With Takeover Requests Affects Drivers’ Takeover Performance and Automation Trust , 2017, Hum. Factors.

[21]  W. Velicer.  Determining the number of components from the matrix of partial correlations , 1976 .

[22]  Alexandra Neukum,et al.  Improving Usefulness of Automated Driving by Lowering Primary Task Interference through HMI Design , 2017 .

[23]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[24]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[25]  Masooda Bashir,et al.  Trust in Automation , 2015, Hum. Factors.

[26]  B P O'Connor,et al.  SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[27]  C. B. Colby.  The weirdest people in the world , 1973 .

[28]  Sebastian Hergeth,et al.  Learning and Development of Mental Models during Interactions with Driving Automation: A Simulator Study , 2019, Proceedings of the 10th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design: driving assessment 2019.

[29]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[30]  Andras Varhelyi,et al.  How Is Acceptance Measured? : Overview of Measurement Issues, Methods and Tools , 2014 .

[31]  Frédéric Vanderhaegen,et al.  Can dissonance engineering improve risk analysis of human–machine systems? , 2017, Cognition, Technology & Work.

[32]  Jakob Nielsen,et al.  Measuring usability: preference vs. performance , 1994, CACM.

[33]  Patricia S. Jones,et al.  An Adaptation of Brislin’s Translation Model for Cross-cultural Research , 2001, Nursing research.

[34]  James R. Lewis,et al.  Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies , 2002, Int. J. Hum. Comput. Interact..

[35]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[36]  Markus Bühner.  Einführung in die Test- und Fragebogenkonstruktion , 2008 .

[37]  Martin Schrepp,et al.  Construction and Evaluation of a User Experience Questionnaire , 2008, USAB.

[38]  B. Reimer,et al.  People must retain control of autonomous vehicles , 2018, Nature.

[39]  Katia P. Sycara,et al.  Towards the Development of an Inter-cultural Scale to Measure Trust in Automation , 2014, HCI.

[40]  Alexandra Neukum,et al.  Increasing anthropomorphism and trust in automated driving functions by adding speech output , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[41]  Klaus Bengler,et al.  Trust in Automation – Before and After the Experience of Take-over Scenarios in a Highly Automated Vehicle☆ , 2015 .

[42]  Nadja Schömig,et al.  Test procedure for evaluating the human–machine interface of vehicles with automated driving systems , 2019, Traffic injury prevention.

[43]  Manfred Tscheligi,et al.  Predicting information technology usage in the car: towards a car technology acceptance model , 2012, AutomotiveUI.

[44]  J. Krems,et al.  Learning and development of trust, acceptance and the mental model of ACC. A longitudinal on-road study , 2015 .

[45]  N. Epley,et al.  The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle , 2014 .

[46]  Frederik Naujoks,et al.  Testing Scenarios for Human Factors Research in Level 3 Automated Vehicles , 2017 .

[47]  Michael Burmester,et al.  AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität , 2003, MuC.

[48]  J. Henrich,et al.  The weirdest people in the world? , 2010, Behavioral and Brain Sciences.

[49]  Thomas S. Tullis,et al.  Readability of fonts in the windows environment , 1995, CHI 95 Conference Companion.

[50]  Fred D. Davis.  A technology acceptance model for empirically testing new end-user information systems : theory and results , 1985 .

[51]  Alexandra Neukum,et al.  Speech improves human-automation cooperation in automated driving , 2016, MuC.

[52]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[53]  Linda Ng Boyle,et al.  Extending the Technology Acceptance Model to assess automation , 2011, Cognition, Technology & Work.

[54]  Paul Green,et al.  PRELIMINARY HUMAN FACTORS DESIGN GUIDELINES FOR DRIVER INFORMATION SYSTEMS. FINAL REPORT , 1995 .

[55]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[56]  Randall D. Spain,et al.  Towards an Empirically Developed Scale for System Trust: Take Two , 2008 .

[57]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004 .

[58]  Michael Weber,et al.  Touch Screen Maneuver Approval Mechanisms for Highly Automated Vehicles: A First Evaluation , 2017, AutomotiveUI.

[59]  Dick de Waard,et al.  A simple procedure for the assessment of acceptance of advanced transport telematics , 1997 .

[60]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[61]  Rainer Stark,et al.  Integrating User Experience Validation into a New Engineering Development Process for Advanced Driver Assistance Systems , 2014, AutomotiveUI.

[62]  Sebastiaan M. Petermeijer,et al.  Usefulness and satisfaction of take-over requests for highly automated driving , 2017 .

[63]  Louis Guttman,et al.  A basis for analyzing test-retest reliability , 1945, Psychometrika.

[64]  William Payre,et al.  Intention to use a fully automated car: attitudes and a priori acceptability , 2014 .

[65]  Emeli Adell,et al.  ACCEPTANCE OF DRIVER SUPPORT SYSTEMS , 2010 .

[66]  Frederik Naujoks,et al.  Use Cases for Assessing, Testing, and Validating the Human Machine Interface of Automated Driving Systems , 2018 .

[67]  Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications , 2016, AutomotiveUI.

[68]  Sebastian Hergeth,et al.  Automation Trust in ConditionalAutomated Driving Systems: Approachesto Operationalization and Design , 2016 .

[69]  Gordon B. Davis,et al.  User Acceptance of Information Technology: Toward a Unified View , 2003, MIS Q..

[70]  M. König,et al.  Users’ resistance towards radical innovations: The case of the self-driving car , 2017 .

[71]  Alexandra Neukum,et al.  Your Turn or My Turn?: Design of a Human-Machine Interface for Conditional Automation , 2016, AutomotiveUI.

[72]  C. Fornell,et al.  Evaluating structural equation models with unobservable variables and measurement error. , 1981 .

[73]  W. Revelle,et al.  Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma , 2009 .

[74]  Riender Happee,et al.  Conceptual Model to Explain, Predict, and Improve User Acceptance of Driverless Podlike Vehicles , 2016 .

[75]  Andreas Butz,et al.  SupportingTrust in Autonomous Driving , 2017, IUI.

[76]  James R. Lewis.  Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies , 2002 .

[77]  Olaf Köller,et al.  Umgang mit fehlenden Werten in der psychologischen Forschung : Probleme und Lösungen , 2007 .

[78]  Linda Ng Boyle,et al.  In UX We Trust: Investigation of Aesthetics and Usability of Driver-Vehicle Interfaces and Their Impact on the Perception of Automated Driving , 2019, CHI.

[79]  S. Levinson,et al.  WEIRD languages have misled us, too , 2010, Behavioral and Brain Sciences.

[80]  Alexandra Neukum,et al.  Controllability of partially automated driving functions - does it matter whether drivers are allowed to take their hands off the steering wheel? , 2015 .

[81]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[82]  A Stevens,et al.  DESIGN GUIDELINES FOR SAFETY OF IN-VEHICLE INFORMATION SYSTEMS , 2002 .

[83]  Frederik Naujoks,et al.  How Usability Can Save the Day - Methodological Considerations for Making Automated Driving a Success Story , 2018, AutomotiveUI.

[84]  Frederik Naujoks,et al.  Towards guidelines and verification methods for automated vehicle HMIs , 2019, Transportation Research Part F: Traffic Psychology and Behaviour.

[85]  Alexandra Fort,et al.  Automotive HMI design and participatory user involvement: review and perspectives , 2017, Ergonomics.

[86]  Dot Hs.  Crash Avoidance Metrics Partnership , 2002 .

[87]  Véronique Sébille,et al.  Sample size used to validate a scale: a review of publications on newly-developed patient reported outcomes measures , 2014, Health and Quality of Life Outcomes.

[88]  Jaap Ham,et al.  Trust in Smart Systems , 2012, Hum. Factors.

[89]  Jan C Zoellick,et al.  Assessing acceptance of electric automated vehicles after exposure in a realistic traffic environment , 2019, PloS one.