“Stop over there”: natural gesture and speech interaction for non-critical spontaneous intervention in autonomous driving

We propose a new multimodal input technique for Non-critical Spontaneous Situations (NCSSs) in autonomous driving scenarios such as selecting a parking lot or picking up a hitchhiker. Speech and deictic (pointing) gestures were combined to instruct the car about desired interventions which include spatial references to the current environment (e.g., ''stop over [pointing] there'' or ''take [pointing] this parking lot''). In this way, advantages from both modalities were exploited: Speech allows for selecting from many maneuvres and functions in the car (e.g., stop, park), whereas deictic gestures provide a natural and intuitive way of indicating spatial discourse referents used in these interventions (e.g., near this tree, that parking lot). The speech and pointing gesture input was compared to speech and touch-based input in a user study with 38 participants. The touch-based input was selected as a baseline due to its widespread use in in-car touch screens. The evaluation showed that speech and pointing gestures are perceived more natural, intuitive and less cognitively demanding compared to speech and touch and are thus recommended as NCSSs intervention technique for autonomous driving.

[1]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[2]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[3]  Dennis Wixon,et al.  The Natural User Interface , 2011 .

[4]  Bernhard Preim,et al.  User Interface Engineering: Einleitung , 2015 .

[5]  Alexandra Neukum,et al.  Your Turn or My Turn?: Design of a Human-Machine Interface for Conditional Automation , 2016, AutomotiveUI.

[6]  Philippe A. Palanque,et al.  Fusion engines for multimodal input: a survey , 2009, ICMI-MLMI '09.

[7]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[8]  James Bucanek.  Model-View-Controller Pattern , 2009 .

[9]  John R. Treat,et al.  Tri-Level Study of the Causes of Traffic Accidents: An overview of final results , 1977 .

[10]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[11]  Michael Weber,et al.  Towards Cooperative Driving: Involving the Driver in an Autonomous Vehicle's Decision Making , 2016, AutomotiveUI.

[12]  James A. Larson,et al.  Guidelines for multimodal user interface design , 2004, CACM.

[13]  Andreas Butz,et al.  Freehand vs. micro gestures in the car: Driving performance and user experience , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[14]  Andreas Butz,et al.  Culturally Independent Gestures for In-Car Interactions , 2013, INTERACT.

[15]  Klaus Bengler,et al.  How Traffic Situations and Non-Driving Related Tasks Affect the Take-Over Quality in Highly Automated Driving , 2014 .

[16]  Murat Yener,et al.  Model View Controller Pattern , 2014 .

[17]  Matthew Turk,et al.  Multimodal interaction: A review , 2014, Pattern Recognit. Lett..

[18]  Marc Erich Latoschik.  A General Framework for Multimodal Interaction in Virtual Reality Systems: PrOSA , 2001 .

[19]  Klaus Bengler,et al.  The ergonomic value of a bidirectional haptic interface when driving a highly automated vehicle , 2013, Cognition, Technology & Work.

[20]  Taku Komura,et al.  Topology matching for fully automatic similarity estimation of 3D shapes , 2001, SIGGRAPH.

[21]  Andreas Butz,et al.  Free-hand pointing for identification and interaction with distant objects , 2013, AutomotiveUI.

[22]  Marc Erich Latoschik,et al.  Semantic Reflection for Intelligent Virtual Environments , 2007, 2007 IEEE Virtual Reality Conference.

[23]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[24]  Arne Jönsson,et al.  Wizard of Oz studies: why and how , 1993, IUI '93.

[25]  Stephen M. Casner,et al.  The challenges of partially automated driving , 2016, Commun. ACM.

[26]  Ralph Bruder,et al.  User acceptance of cooperative maneuver-based driving--a summary of three studies. , 2012, Work.

[27]  James Bucanek.  Learn Objective-C for Java Developers , 2009 .

[28]  Marc Erich Latoschik.  A user interface framework for multimodal VR interactions , 2005, ICMI '05.

[29]  M R Endsley,et al.  Level of automation effects on performance, situation awareness and workload in a dynamic control task. , 1999, Ergonomics.

[30]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[31]  Albrecht Schmidt,et al.  Multimodal interaction in the car: combining speech and gestures on the steering wheel , 2012, AutomotiveUI.

[32]  Timothy Brittain-Catlin.  Put it there , 2013 .

[33]  Marc Erich Latoschik,et al.  Exploiting Distant Pointing Gestures for Object Selection in a Virtual Environment , 1997, Gesture Workshop.

[34]  Elena Mugellini,et al.  A comparison of three interaction modalities in the car: gestures, voice and touch , 2016, IHM.

[35]  Marc Erich Latoschik,et al.  Resolving object references in multimodal dialogues for immersive virtual environments , 2004, IEEE Virtual Reality 2004.

[36]  Marc Erich Latoschik.  Designing transition networks for multimodal VR-interactions using a markup language , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[37]  Mark Vollrath,et al.  Accident Analysis and Prevention , 2009 .

[38]  Martin Fischbach,et al.  Software Techniques for Multimodal Input Processing in Realtime Interactive Systems , 2015, ICMI.

[39]  Raúl Rojas,et al.  Semi-autonomous Car Control Using Brain Computer Interfaces , 2012, IAS.

[40]  Catherine M. Burns,et al.  Autonomous Driving in the Real World: Experiences with Tesla Autopilot and Summon , 2016, AutomotiveUI.

[41]  Jörn Hurtienne,et al.  Intuitive Use of User Interfaces: Defining a Vague Concept , 2007, HCI.

[42]  Antoine Raux,et al.  The HRI-CMU Corpus of Situated In-Car Interactions , 2016 .

[43]  K. Bengler,et al.  Literaturanalyse und Methodenauswahl zur Gestaltung von Systemen zum hochautomatisierten Fahren , 2015 .