Human-Vehicle Interaction to Support Driver's Situation Awareness in Automated Vehicles: A Systematic Review

Autonomous driving will change the role of the driver. From being the main actor in driving, the driver will now have a supervisory role during the autonomous driving phases. However, if the driver has to take over control of the vehicle, he must be aware of the situation around him. This is why it is important to develop interfaces to keep him in the loop. This article proposes a systematic review of Human Vehicle Interaction providing situation awareness in the context of autonomous driving. Thirty seven articles presenting such interactions are analyzed in terms of design of the interaction (modalities, location, conveyed information) but also in term of evaluation and experimental conditions. We present an overview of previous studies in order to highlight the work already done or in progress. Current studies present mainly monomodal interfaces although the evaluation of multimodal interactions present promising results in this field.

[1]  K. Kunze,et al.  ThermalDrive - Towards Situation Awareness over Thermal Feedback in Automated Driving Scenarios , 2022, IUI Companion.

[2]  Stefan Schneegass,et al.  Towards Transparent Behavior of Automated Vehicles: Design and Evaluation of HUD Concepts to Support System Predictability Through Motion Intent Communication , 2021, MobileHCI.

[3]  Yee Mun Lee,et al.  Don't Worry, I'm in Control! Is Users’ Trust in Automated Driving Different When Using a Continuous Ambient Light HMI Compared to an Auditory HMI? , 2021, AutomotiveUI.

[4]  Dustin J. Souders,et al.  Age Differences in the Malleability of Attitudes toward Automated Shared Mobility , 2021, Proceedings of the Human Factors and Ergonomics Society Annual Meeting.

[5]  Mark Colley,et al.  Effects of Semantic Segmentation Visualization on Trust, Situation Awareness, and Cognitive Load in Highly Automated Vehicles , 2021, CHI.

[6]  Rebecca M. Currano,et al.  Little Road Driving HUD: Heads-Up Display Complexity Influences Drivers’ Perceptions of Automated Vehicles , 2021, CHI.

[7]  Steve Love,et al.  Increasing the User Experience in Autonomous Driving through different Feedback Modalities , 2021, IUI.

[8]  Andreas Riener,et al.  StickyWSD: Investigating Content Positioning on a Windshield Display for Automated Driving , 2020, MUM.

[9]  A. Korthauer,et al.  Watch Your Vehicle Driving at the City: Interior HMI with Augmented Reality for Automated Driving , 2020, MobileHCI.

[10]  Omar Abou Khaled,et al.  The Effect of Instructions and Context-Related Information about Limitations of Conditionally Automated Vehicles on Situation Awareness , 2020, AutomotiveUI.

[11]  Bruce N. Walker,et al.  Situational Trust Scale for Automated Driving (STS-AD): Development and Initial Validation , 2020, AutomotiveUI.

[12]  Elena Mugellini,et al.  Designing an AI-Companion to Support the Driver in Highly Autonomous Cars , 2020, HCI.

[13]  A. Bosco,et al.  The Drives for Driving Simulation: A Scientometric Analysis and a Selective Review of Reviews on Simulated Driving Research , 2020, Frontiers in Psychology.

[14]  Stewart Birrell,et al.  Designing an Adaptive Interface: Using Eye Tracking to Classify How Information Usage Changes Over Time in Partially Automated Vehicles , 2020, IEEE Access.

[15]  Sabine Langlois,et al.  Principles of transparency for autonomous vehicles: first results of an experiment with an augmented reality human–machine interface , 2019, Cognition, Technology & Work.

[16]  Rebecca A. Grier,et al.  Using a Situational Awareness Display to Improve Rider Trust and Comfort with an AV Taxi , 2019, Proceedings of the Human Factors and Ergonomics Society Annual Meeting.

[17]  Seung-Jun Lee,et al.  A study on the Effects of Providing Situation Awareness Information for the Control Authority Transition of Automated Vehicle , 2019, 2019 International Conference on Information and Communication Technology Convergence (ICTC).

[18]  Mica R Endsley,et al.  A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM , 2019, Hum. Factors.

[19]  David R. Large,et al.  A Longitudinal Simulator Study to Explore Drivers' Behaviour in Level 3 Automated Vehicles , 2019, AutomotiveUI.

[20]  Alexander Kunze,et al.  Conveying Uncertainties Using Peripheral Awareness Displays in the Context of Automated Driving , 2019, AutomotiveUI.

[21]  Elena Mugellini,et al.  Owner Manuals Review and Taxonomy of ADAS Limitations in Partially Automated Vehicles , 2019, AutomotiveUI.

[22]  Heinrich Hußmann,et al.  I Drive - You Trust: Explaining Driving Behavior Of Autonomous Cars , 2019, CHI Extended Abstracts.

[23]  Tobias Vogelpohl,et al.  Asleep at the automated wheel-Sleepiness and fatigue during highly automated driving. , 2019, Accident; analysis and prevention.

[24]  Helmut Krcmar,et al.  Improving take-over quality in automated driving by interrupting non-driving tasks , 2019, IUI.

[25]  Dawn M. Tilbury,et al.  Situational Awareness, Drivers Trust in Automated Driving Systems and Secondary Task Performance , 2019, SAE International Journal of Connected and Automated Vehicles.

[26]  Klaus Bengler,et al.  An HMI Concept to Improve Driver's Visual Behavior and Situation Awareness in Automated Vehicle , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[27]  Gerhard Rigoll,et al.  Catch My Drift: Elevating Situation Awareness for Highly Automated Driving with an Explanatory Windshield Display User Interface , 2018, Multimodal Technol. Interact..

[28]  Marianna Obrist,et al.  I Smell Trouble: Using Multiple Scents To Convey Driving-Relevant Information , 2018, ICMI.

[29]  Wendy Ju,et al.  Don't Be Alarmed: Sonifying Autonomous Vehicle Perception to Increase Situation Awareness , 2018, AutomotiveUI.

[30]  Stefan Schneegaß,et al.  Design Guidelines for Reliability Communication in Autonomous Vehicles , 2018, AutomotiveUI.

[31]  Heinrich Hußmann,et al.  Adjunct Proceedings of the 10th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications , 2018 .

[32]  Heinrich H. Bülthoff,et al.  Feel the Movement: Real Motion Influences Responses to Take-over Requests in Highly Automated Vehicles , 2018, CHI.

[33]  Chee Peng Lim,et al.  A Review of Situation Awareness Assessment Approaches in Aviation Environments , 2018, IEEE Systems Journal.

[34]  Elisabeth Schmidt,et al.  Mitigating passive fatigue during monotonous drives with thermal stimuli: Insights into the effect of different stimulation durations. , 2017, Accident; analysis and prevention.

[35]  Wendy Ju,et al.  Reinventing the Wheel: Transforming Steering Wheel Systems for Autonomous Vehicles , 2017, UIST.

[36]  Satoshi Okamoto,et al.  Anthropomorphic AI Agent Mediated Multimodal Interactions in Vehicles , 2017, AutomotiveUI.

[37]  Matthias Rauterberg,et al.  Experimental Setup of Motion Sickness and Situation Awareness in Automated Vehicle Riding Experience , 2017, AutomotiveUI.

[38]  Jacques M. B. Terken,et al.  Situation Awareness in Automated Vehicles through Proximal Peripheral Light Signals , 2017, AutomotiveUI.

[39]  Lena Rittger,et al.  Guiding Driver Visual Attention with LEDs , 2017, AutomotiveUI.

[40]  Matthias Rauterberg,et al.  Situation Awareness and Motion Sickness in Automated Vehicle Driving Experience: A Preliminary Study of Peripheral Visual Information , 2017, AutomotiveUI.

[41]  Jun Hu,et al.  Designing for Enhancing Situational Awareness of Semi-Autonomous Driving Vehicles , 2017, AutomotiveUI.

[42]  Takahiro Wada,et al.  Displaying System Situation Awareness Increases Driver Trust in Automated Driving , 2017, IEEE Transactions on Intelligent Vehicles.

[43]  Alexandra Neukum,et al.  Improving Usefulness of Automated Driving by Lowering Primary Task Interference through HMI Design , 2017 .

[44]  Alexandra Neukum,et al.  A Review of Non-driving-related Tasks Used in Studies on Automated Driving , 2017 .

[45]  R. M. Taylor,et al.  Situational Awareness Rating Technique (Sart): The Development of a Tool for Aircrew Systems Design , 2017 .

[46]  Jinwoo Kim,et al.  "Are You Ready to Take-over?": An Exploratory Study on Visual Assistance to Enhance Driver Vigilance , 2017, CHI Extended Abstracts.

[47]  Wendy Ju,et al.  Toward Measurement of Situation Awareness in Autonomous Vehicles , 2017, CHI.

[48]  Fang Chen,et al.  Using Advisory 3D Sound Cues to Improve Drivers' Performance and Situation Awareness , 2017, CHI.

[49]  Renaud Deborne,et al.  Transition of control in a partially automated vehicle: Effects of anticipation and non-driving-related task involvement , 2017 .

[50]  Arie P. van den Beukel,et al.  How to assess driver's interaction with partially automated driving systems - A framework for early concept assessment. , 2017, Applied ergonomics.

[51]  Monica Bordegoni,et al.  Demonstrating the Effectiveness of Olfactory Stimuli on Drivers’ Attention , 2017 .

[52]  Louis Tijerina,et al.  An Exploratory Study of Driver Response to Reduced System Confidence Notifications in Automated Driving , 2016, IEEE Transactions on Intelligent Vehicles.

[53]  Ronald Schroeter,et al.  Pokémon DRIVE: towards increased situational awareness in semi-automated driving , 2016, OZCHI.

[54]  Susanne Boll,et al.  AutoAmbiCar: Using Ambient Light to Inform Drivers About Intentions of Their Automated Cars , 2016, AutomotiveUI.

[55]  Enrico Rukzio,et al.  Carvatar: Increasing Trust in Highly-Automated Driving Through Social Cues , 2016, AutomotiveUI.

[56]  Saskia Bakker,et al.  The Interaction-Attention Continuum: Considering Various Levels of Human Attention in Interaction Design , 2016 .

[57]  Michael A. Nees,et al.  Speech Auditory Alerts Promote Memory for Alerted Events in a Video-Simulated Self-Driving Car Ride , 2016, Hum. Factors.

[58]  Poong Hyun Seong,et al.  A Quantitative Team Situation Awareness Measurement Method Considering Technical and Nontechnical Skills of Teams , 2016 .

[59]  Omer Tsimhoni,et al.  Haptic seat for automated driving: preparing the driver to take control effectively , 2015, AutomotiveUI.

[60]  Wendy Ju,et al.  DAZE: a real-time situation awareness measurement tool for driving , 2015, AutomotiveUI.

[61]  Frank Gauterin,et al.  Recommendations Supporting Situation Awareness in Partially Automated Driver Assistance Systems , 2015, IEEE Transactions on Intelligent Transportation Systems.

[62]  Isabelle Milleville-Pennel,et al.  Driving for Real or on a Fixed-Base Simulator: Is It so Different? An Explorative Study , 2015, PRESENCE: Teleoperators and Virtual Environments.

[63]  Niklas Strand,et al.  Semi-automated versus highly automated driving in critical situations caused by automation failures , 2014 .

[64]  Natasha Merat,et al.  Transition to manual: driver behaviour when resuming control from a highly automated vehicle , 2014 .

[65]  Lynne Baillie,et al.  What's around the corner?: enhancing driver awareness in autonomous vehicles via in-vehicle spatial auditory displays , 2014, NordiCHI.

[66]  Frank Lai,et al.  Human Factors of Highly Automated Driving: Results from the EASY and CityMobil Projects , 2014 .

[67]  Kathrin Zeeb,et al.  Effects of human-machine interaction mechanisms on situation awareness in partly automated driving , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[68]  Klaus Bengler,et al.  Requirements for cooperative vehicle guidance , 2013, 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013).

[69]  Klaus Bengler,et al.  “Take over!” How long does it take to get the driver back into the loop? , 2013 .

[70]  Qun Wang,et al.  An experimental analysis of situation awareness for cockpit display interface evaluation based on flight simulation , 2013 .

[71]  Mark Vollrath,et al.  Improving the Driver–Automation Interaction , 2013, Hum. Factors.

[72]  Israel Abramov,et al.  Sex and vision II: color appearance of monochromatic lights , 2012, Biology of Sex Differences.

[73]  Israel Abramov,et al.  Sex & vision I: Spatio-temporal resolution , 2012, Biology of Sex Differences.

[74]  Neville A. Stanton,et al.  Situation awareness on the road: review, theoretical and methodological issues, and future directions , 2012 .

[75]  Natasha Merat,et al.  Highly Automated Driving, Secondary Task Performance, and Driver State , 2012, Hum. Factors.

[76]  Rolf Zon,et al.  Eye Movements as an Indicator of Situation Awareness in a Flight Simulator Experiment , 2012 .

[77]  Alberto Megías,et al.  Effect of Directional Speech Warnings on Road Hazard Detection , 2011, Traffic injury prevention.

[78]  Jason Bennett Thatcher,et al.  Trust in a specific technology: An investigation of its components and measures , 2011, TMIS.

[79]  Albrecht Schmidt,et al.  Design space for driver-based automotive user interfaces , 2009, AutomotiveUI.

[80]  Martin Schrepp,et al.  Construction and Evaluation of a User Experience Questionnaire , 2008, USAB.

[81]  Tara Matthews,et al.  Defining, Designing, and Evaluating Peripheral Displays: An Analysis Using Activity Theory , 2007, Hum. Comput. Interact..

[82]  Neville A Stanton,et al.  Back to the future: Brake reaction times for manual and automated vehicles , 2007, Ergonomics.

[83]  John T. Stasko,et al.  A taxonomy of ambient information systems: four patterns of design , 2006, AVI '06.

[84]  J. Brabyn,et al.  Night Driving Self-Restriction: Vision Function and Gender Differences , 2005, Optometry and vision science : official publication of the American Academy of Optometry.

[85]  Jean Scholtz,et al.  Implementation of a situation awareness assessment tool for evaluation of human-robot interfaces , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[86]  Tara Matthews,et al.  A toolkit for managing user attention in peripheral displays , 2004, UIST '04.

[87]  Susan J. Rottunda,et al.  The Influence of Age and Gender on the Driving Patterns of Older Adults , 2003, Journal of women & aging.

[88]  Lawrence G. Shattuck,et al.  Situation Awareness for Military Ground Forces: Current Issues and Perspectives , 2001 .

[89]  Michael A. Vidulich,et al.  The Relationship between Mental Workload and Situation Awareness , 2000 .

[90]  Fred D. Davis,et al.  A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies , 2000, Management Science.

[91]  Thomas D. Hardiman,et al.  A Comparative Analysis of Sagat and Sart for Evaluations of Situation Awareness , 1998 .

[92]  Dick de Waard,et al.  A simple procedure for the assessment of acceptance of advanced transport telematics , 1997 .

[93]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[94]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[95]  Kip Smith,et al.  Situation Awareness Is Adaptive, Externally Directed Consciousness , 1995, Hum. Factors.

[96]  Mica R. Endsley,et al.  Situation awareness global assessment technique (SAGAT) , 1988, Proceedings of the IEEE 1988 National Aerospace and Electronics Conference.

[97]  Markus Lienkamp,et al.  Driving in an Increasingly Automated World – Approaches to Improve the Driver-automation Interaction , 2015 .

[98]  María José Abásolo Guerrero,et al.  Improving Situational Awareness in Military Operations using Augmented Reality , 2015, WSCG 2015.

[99]  Stephanie Arndt,et al.  Evaluierung der Akzeptanz von Fahrerassistenzsystemen , 2011 .

[100]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[101]  G. Bedny,et al.  Theory of Activity and Situation Awareness , 1999 .

[102]  M. Endsley Situation Awareness In Aviation Systems , 1999 .

[103]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[104]  Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles , 2022 .