Effects of Semantic Segmentation Visualization on Trust, Situation Awareness, and Cognitive Load in Highly Automated Vehicles

Autonomous vehicles could improve mobility, safety, and inclusion in traffic. While this technology seems within reach, its successful introduction depends on the intended user’s acceptance. A substantial factor for this acceptance is trust in the autonomous vehicle’s capabilities. Visualizing internal information processed by an autonomous vehicle could calibrate this trust by enabling the perception of the vehicle’s detection capabilities (and its failures) while only inducing a low cognitive load. Additionally, the simultaneously raised situation awareness could benefit potential take-overs. We report the results of two comparative online studies on visualizing semantic segmentation information for the human user of autonomous vehicles. Effects on trust, cognitive load, and situation awareness were measured using a simulation (N=32) and state-of-the-art panoptic segmentation on a pre-recorded real-world video (N=41). Results show that the visualization using Augmented Reality increases situation awareness while remaining low cognitive load.

[1]  Anna Schieben,et al.  Reflecting the automated vehicle's perception and intention: Light-based interaction approaches for on-board HMI in highly automated vehicles , 2020, IUI Companion.

[2]  Jian Yang,et al.  Importance-Aware Semantic Segmentation for Autonomous Vehicles , 2019, IEEE Transactions on Intelligent Transportation Systems.

[3]  Neville A. Stanton,et al.  Effects of adaptive cruise control and highly automated driving on workload and situation awareness: A review of the empirical evidence , 2014 .

[4]  Andreas Butz,et al.  Supporting Trust in Autonomous Driving , 2017 .

[5]  Thomas S. Huang,et al.  Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Bastian Pfleging,et al.  Investigating user needs for non-driving-related activities during automated driving , 2016, MUM.

[7]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[8]  Stefan Brandenburg,et al.  Switching from manual to automated driving and reverse: Are drivers behaving more risky after highly automated driving? , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[9]  Daniel J. Fagnant,et al.  Preparing a Nation for Autonomous Vehicles: Opportunities, Barriers and Policy Recommendations , 2015 .

[10]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Moritz Körber,et al.  Theoretical Considerations and Development of a Questionnaire to Measure Trust in Automation , 2018, Advances in Intelligent Systems and Computing.

[12]  R. M. Taylor,et al.  Situational Awareness Rating Technique (Sart): The Development of a Tool for Aircrew Systems Design , 2017 .

[13]  Joseph L. Gabbard,et al.  Behind the Glass: Driver Challenges and Opportunities for AR Automotive Applications , 2014, Proceedings of the IEEE.

[14]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[15]  Johannes Kraus,et al.  How to Design Valid Simulator Studies for Investigating User Experience in Automated Driving: Review and Hands-On Considerations , 2018, AutomotiveUI.

[16]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[17]  Alexander Kunze,et al.  Augmented Reality Displays for Communicating Uncertainty Information in Automated Driving , 2018, AutomotiveUI.

[18]  Gabriel J. Brostow,et al.  Digging Into Self-Supervised Monocular Depth Estimation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Michael Sivak,et al.  A Survey of Public Opinion about Autonomous and Self-Driving Vehicles in the U.S., the U.K., and Australia , 2014 .

[20]  Susanne Boll,et al.  AutoAmbiCar: Using Ambient Light to Inform Drivers About Intentions of Their Automated Cars , 2016, AutomotiveUI.

[21]  Joseph L. Gabbard,et al.  Head-Up vs. Head-Down Displays: Examining Traditional Methods of Display Assessment While Driving , 2016, AutomotiveUI.

[22]  R. Rosenthal.  Parametric measures of effect size. , 1994 .

[23]  Philipp Wintersberger,et al.  Fostering User Acceptance and Trust in Fully Automated Vehicles: Evaluating the Potential of Augmented Reality , 2018, PRESENCE: Virtual and Augmented Reality.

[24]  Edgar Brunner,et al.  Nonparametric analysis of longitudinal data in factorial experiments , 2012 .

[25]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[26]  Wendy Ju,et al.  Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance , 2014, International Journal on Interactive Design and Manufacturing (IJIDeM).

[27]  Thomas D. Hardiman,et al.  A Comparative Analysis of Sagat and Sart for Evaluations of Situation Awareness , 1998 .

[28]  Mark Colley,et al.  Effect of Visualization of Pedestrian Intention Recognition on Trust and Cognitive Load , 2020, AutomotiveUI.

[29]  Göran Falkman,et al.  Presenting system uncertainty in automotive UIs for supporting trust calibration in autonomous driving , 2013, AutomotiveUI.

[30]  Timothy D. Wilson,et al.  The halo effect: Evidence for unconscious alteration of judgments. , 1977 .

[31]  Johannes Kraus,et al.  The More You Know: Trust Dynamics and Calibration in Highly Automated Driving and the Effects of Take-Overs, System Malfunction, and System Transparency , 2019, Hum. Factors.

[32]  Johannes Kraus,et al.  Towards Opt-Out Permission Policies to Maximize the Use of Automated Driving , 2019, AutomotiveUI.

[33]  Natasha Merat,et al.  Transition to manual: driver behaviour when resuming control from a highly automated vehicle , 2014 .

[34]  Riender Happee,et al.  Public opinion on automated driving: results of an international questionnaire among 5000 respondents , 2015 .

[35]  N. Moray,et al.  Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. , 1996, Ergonomics.

[36]  S. Page.  Uncertainty, Difficulty, and Complexity , 2008 .

[37]  Mark R. Lehto,et al.  Foundations for an Empirically Determined Scale of Trust in Automated Systems , 2000 .

[38]  Alexandra Neukum,et al.  Standardized Test Procedure for External Human-Machine Interfaces of Automated Vehicles , 2020, Inf..

[39]  Gerhard Rigoll,et al.  Catch My Drift: Elevating Situation Awareness for Highly Automated Driving with an Explanatory Windshield Display User Interface , 2018, Multimodal Technol. Interact..

[40]  Sigmar-Olaf Tergan,et al.  Visualizing Knowledge and Information: An Introduction , 2005, Knowledge and Information Visualization.

[41]  Alexander Kunze,et al.  Conveying Uncertainties Using Peripheral Awareness Displays in the Context of Automated Driving , 2019, AutomotiveUI.

[42]  Jacob Haspiel,et al.  Look Who's Talking Now: Implications of AV's Explanations on Driver's Trust, AV Preference, Anxiety and Mental Workload , 2019, Transportation Research Part C: Emerging Technologies.

[43]  Andreas Butz,et al.  First Steps towards a View Management Concept for Large-sized Head-up Displays with Continuous Depth , 2016, AutomotiveUI.

[44]  谢启堂,et al.  Head-up display , 2014 .