Design and evaluation of auditory-supported air gesture controls in vehicles

Using touchscreens while driving introduces competition for visual attention that increases crash risk. To resolve this issue, we have developed an auditory-supported air gesture system. We conducted two experiments using the driving simulator to investigate the influence of this system on driving performance, eye glance behavior, secondary task performance, and driver workload. In Experiment 1 we investigated the impact of menu layout and auditory displays with 23 participants. In Experiment 2 we compared the best systems from Experiment 1 with equivalent touchscreen systems with 24 participants. Results from Experiment 1 showed that menus arranged in 2 × 2 grids outperformed systems with 4 × 4 grids across all measures and also demonstrated that auditory displays can be used to reduce visual demands of in-vehicle controls. In Experiment 2 auditory-supported air gestures allowed drivers to look at the road more, showed equivalent driver workload and driving performance, and slightly decreased secondary task performance compared to touchscreens. Implications are discussed with multiple resources theory and Fitts’s law.

[1]  Matthew Heath,et al.  Augmented feedback influences upper limb reaching movement times but does not explain violations of Fitts' Law , 2015, Front. Psychol..

[2]  I. Scott MacKenzie,et al.  Fitts' Law as a Research and Design Tool in Human-Computer Interaction , 1992, Hum. Comput. Interact..

[3]  Karlene K. Ball,et al.  Advances in Technology Used to Assess and Retrain Older Drivers , 2002 .

[4]  Linda Ng Boyle,et al.  Differences in Off-Road Glances: Effects on Young Drivers’ Performance , 2010 .

[5]  Stephen Brewster.  Sonically-enhanced drag and drop , 1998 .

[6]  Ravin Balakrishnan,et al.  Fitts' law and expanding targets: Experimental studies and designs for user interfaces , 2005, TCHI.

[7]  Steven W. Keele,et al.  Movement control in skilled motor performance. , 1968 .

[8]  Linda Ng Boyle,et al.  Driver's lane keeping ability with eyes off road: Insights from a naturalistic study. , 2013, Accident; analysis and prevention.

[9]  C. Wickens,et al.  Situation Awareness, Mental Workload, and Trust in Automation: Viable, Empirically Supported Cognitive Engineering Constructs , 2008 .

[10]  Mark Billinghurst,et al.  A user study of auditory versus visual interfaces for use while driving , 2008, Int. J. Hum. Comput. Stud..

[11]  Bruce N. Walker,et al.  A Multimodal Air Gesture Interface for In Vehicle Menu Navigation , 2014, AutomotiveUI.

[12]  Paul Green,et al.  Driver Distraction, Telematics Design, and Workload Managers: Safety Issues and Solutions , 2004 .

[13]  Abdulmotaleb El-Saddik,et al.  Motion-path based in car gesture control of the multimedia devices , 2011, DIVANet '11.

[14]  Klaus Bengler,et al.  Gesture Control for Use in Automobiles , 2000, MVA.

[15]  Xiaolong Wu,et al.  Choosing the Right Air Gesture: Impacts of Menu Length and Air Gesture Type on Driver Workload , 2016, AutomotiveUI.

[16]  Thomas A. Dingus,et al.  TRAVTEK EVALUATION TASK C3 CAMERA CAR STUDY , 1995 .

[17]  Dot Hs.  Text Reading and Text Input Assessment in Support of the NHTSA Visual-Manual Driver Distraction Guidelines , 2013 .

[18]  E. G. Aiken,et al.  Memory processes in the identification of pitch , 1974 .

[19]  W. Helsen,et al.  A century later: Woodworth's (1899) two-component model of goal-directed aiming. , 2001, Psychological bulletin.

[20]  Mihaela A. Zahariev,et al.  Auditory contact cues improve performance when grasping augmented and virtual objects with a tool , 2008, Experimental Brain Research.

[21]  Romeo Chua,et al.  On-line control of rapid aiming movements: Unexpected target perturbations and movement kinematics. , 1998 .

[22]  P. Fitts,et al.  INFORMATION CAPACITY OF DISCRETE MOTOR RESPONSES. , 1964, Journal of experimental psychology.

[23]  E. G. Aiken,et al.  Memory for the pitch of a tone , 1966 .

[24]  J. Hoffman,et al.  The role of visual attention in saccadic eye movements , 1995, Perception & psychophysics.

[25]  S. A. Wallace,et al.  Visual Control of Discrete Aiming Movements , 1983, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[26]  Gavriel Salvendy,et al.  Modelling of Menu Design in Computerized Work , 1995, Interact. Comput..

[27]  M Akamatsu,et al.  Please Scroll down for Article Ergonomics a Comparison of Tactile, Auditory, and Visual Feedback in a Pointing Task Using a Mouse-type Device , 2022 .

[28]  Stephen A. Brewster,et al.  The design of sonically-enhanced widgets , 1998, Interact. Comput..

[29]  Brittany Anne Klein.  Development of a Statistical Model of Reach Exertion Magnitude Perception for Use in Designing Truck Cabs , 2012 .

[30]  Omer Tsimhoni,et al.  Visual Demand of Driving and the Execution of Display-Intensive in-Vehicle Tasks , 2001 .

[31]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[32]  Terri L. Bonebright,et al.  Testing the effectiveness of sonified graphs for education: A programmatic research project , 2001 .

[33]  Christopher D. Wickens,et al.  Examining the Impact of Cell Phone Conversations on Driving Using Meta-Analytic Techniques , 2006, Hum. Factors.

[34]  Bruce N. Walker,et al.  Auditory Displays for In-Vehicle Technologies: , 2011 .

[35]  Les G. Carlton,et al.  Visual Information: The Control of Aiming Movements , 1981 .

[36]  L Parmer Tijerina.  DRIVER WORKLOAD ASSESSMENT OF ROUTE GUIDANCE SYSTEM DESTINATION ENTRY WHILE DRIVING: A TEST TRACK STUDY , 1998 .

[37]  Jindong Ren,et al.  Seated reach capabilities for ergonomic design and evaluation with consideration of reach difficulties. , 2017, Applied ergonomics.

[38]  Stanley R. Parkinson,et al.  Computer display menus , 1983 .

[39]  D. Roth,et al.  Relation of Useful Field of View and other Screening Tests to on-Road Driving Performance , 2000, Perceptual and motor skills.

[40]  Steve Hansen,et al.  Visual regulation of manual aiming: A comparison of methods , 2010, Behavior research methods.

[41]  Linda R. Elliott,et al.  Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis , 2006, ICMI '06.

[42]  Dot Hs,et al.  The 100-Car Naturalistic Driving Study Phase II - Results of the 100-Car Field Experiment , 2006 .

[43]  I. Scott MacKenzie,et al.  Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts' law research in HCI , 2004, Int. J. Hum. Comput. Stud..

[44]  Paul Green,et al.  Evaluation of a driver interface: effects of control type (knob versus buttons) and menu structure (depth versus breadth) , 1997 .

[45]  H Alm,et al.  The effects of a mobile telephone task on driver behaviour in a car following situation. , 1995, Accident; analysis and prevention.

[46]  James S. Sims,et al.  Moment-theory investigations of photoabsorption and dispersion profiles in atoms and ions , 1976 .

[47]  Myounghoon Jeon,et al.  Influences of visual and auditory displays on aimed movements using air gesture controls , 2017 .

[48]  Bruce N. Walker,et al.  The Effect of a Speech Discrimination Task on Navigation in a Virtual Environment , 2006 .

[49]  Ravin Balakrishnan,et al.  "Beating" Fitts' law: virtual enhancements for pointing facilitation , 2004, Int. J. Hum. Comput. Stud..

[50]  Andreas Riener.  Gestural Interaction in Vehicular Applications , 2012, Computer.

[51]  Bruce N. Walker,et al.  Exploring and evaluating the capabilities of Kinect v2 in a driving simulator environment , 2015, AutomotiveUI.

[52]  Steven Landry,et al.  Towards an in-vehicle sonically-enhanced gesture control interface: A pilot study , 2016, ICAD 2016.

[53]  Omer Tsimhoni,et al.  Address Entry While Driving: Speech Recognition Versus a Touch-Screen Keyboard , 2004, Hum. Factors.

[54]  S. S. Stevens.  A scale for the measurement of a psychological magnitude: loudness. , 1936 .

[55]  Bruce N. Walker,et al.  Comparison of Surface Gestures and Air Gestures for In-Vehicle Menu Navigation , 2016 .

[56]  Myounghoon Jeon,et al.  Spindex (Speech Index) Improves Auditory Menu Acceptance and Navigation Performance , 2011, TACC.

[57]  S. A. Wallace,et al.  Decision time and movement time as a function of response complexity in retarded persons. , 1978, American journal of mental deficiency.

[58]  Raghavan Srinivasan,et al.  Effect of Selected In-Vehicle Route Guidance Systems on Driver Reaction Times , 1997, Hum. Factors.

[59]  Alistair D. N. Edwards,et al.  Soundtrack: An Auditory Interface for Blind Users (Abstract Only) , 1989, SGCH.

[60]  Andy Cockburn,et al.  Multimodal feedback for the acquisition of small targets , 2005, Ergonomics.

[61]  G. Brogmus.  Effects of Age and Sex on Speed and Accuracy of Hand Movements: And the Refinements they Suggest for Fitts' Law , 1991 .

[62]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[63]  M. Boltz,et al.  Tempo discrimination of musical patterns: Effects due to pitch and rhythmic structure , 1998, Perception & psychophysics.

[64]  M. Posner,et al.  Processing of visual feedback in rapid movements. , 1968, Journal of experimental psychology.

[65]  Edward Lank,et al.  Endpoint prediction using motion kinematics , 2007, CHI.

[66]  Christopher D. Wickens,et al.  In-Vehicle Glance Duration , 2007 .

[67]  Louis Tijerina,et al.  MODELLING THE RELATIONSHIP BETWEEN DRIVER IN-VEHICLE VISUAL DEMANS AND ACCIDENT OCCURRENCE , 1998 .

[68]  Judy Edworthy.  Does sound help us to work better with machines? A commentary on Rauterberg's paper ‘About the importance of auditory alarms during the operation of a plant simulator’ , 1998 .

[69]  Frank Drews,et al.  Text Messaging During Simulated Driving , 2009, Hum. Factors.

[70]  Gary E. Burnett,et al.  On-the-move destination entry for vehicle navigation systems: Unsafe by any means? , 2004, Behav. Inf. Technol..

[71]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[72]  Michael A. Regan,et al.  Driver distraction: A review of the literature , 2003 .

[73]  Bruce N Walker,et al.  Magnitude estimation of conceptual data dimensions for use in sonification. , 2002, Journal of experimental psychology. Applied.

[74]  D. Elliott,et al.  Visual regulation of manual aiming , 1993 .

[75]  Dwight P. Miller.  The Depth/Breadth Tradeoff in Hierarchical Computer Menus , 1981 .

[76]  E. B. Newman,et al.  A Scale for the Measurement of the Psychological Magnitude Pitch , 1937 .

[77]  Matthew N. Dailey,et al.  A comparison of audio and tactile displays for non-visual target selection tasks , 2010, 2010 International Conference on User Science and Engineering (i-USEr).

[78]  R. Kerr,et al.  Movement time in an underwater environment. , 1973, Journal of motor behavior.

[79]  Robert Sessions Woodworth,et al.  THE ACCURACY OF VOLUNTARY MOVEMENT , 1899 .

[80]  Carl Gutwin,et al.  A Predictive Model of Human Performance With Scrolling and Hierarchical Lists , 2009, Hum. Comput. Interact..

[81]  W. E. Hick.  Quarterly Journal of Experimental Psychology , 1948, Nature.

[82]  Kirsty Beilharz.  Gesture-controlled interaction with aesthetic information sonification , 2005 .

[83]  Andrea C. Pierno,et al.  Effects of increasing visual load on aurally and visually guided target acquisition in a virtual environment. , 2005, Applied ergonomics.

[84]  Xiaozhi Yu,et al.  Modeling study of seated reach envelopes based on spherical harmonics with consideration of the difficulty ratings. , 2017, Applied ergonomics.

[85]  William R. Wyatt,et al.  Effects of Auditory Feedback on Movement Time in a Fitts Task , 2010, Journal of motor behavior.

[86]  Denis McKeown,et al.  Mapping Candidate Within-Vehicle Auditory Displays to Their Referents , 2007, Hum. Factors.

[87]  Christopher D. Wickens,et al.  Multiple resources and performance prediction , 2002 .

[88]  Thomas A. Dingus,et al.  The Impact of Driver Inattention on Near-Crash/Crash Risk: An Analysis Using the 100-Car Naturalistic Driving Study Data , 2006 .

[89]  William W. Gaver.  The SonicFinder: An Interface That Uses Auditory Icons , 1989, Hum. Comput. Interact..

[90]  Yoshitaka Nakajima,et al.  Auditory Scene Analysis: The Perceptual Organization of Sound Albert S. Bregman , 1992 .

[91]  Myounghoon Jeon,et al.  “Spindex”: Accelerated Initial Speech Sounds Improve Navigation Performance in Auditory Menus , 2009 .

[92]  E. Salas,et al.  Human Factors : The Journal of the Human Factors and Ergonomics Society , 2012 .

[93]  R A Abrams,et al.  Optimality in human motor performance: ideal control of rapid aimed movements. , 1988, Psychological review.

[94]  Bruce N. Walker,et al.  Consistency of magnitude estimations with conceptual data dimensions used for sonification , 2007 .

[95]  Panayiotis Zaphiris,et al.  Depth vs Breath in the Arrangement of Web Links , 2000 .

[96]  Paul Green.  INTERNATIONAL INCIDENTS: CRASHES INDUCED BY DRIVER INFORMATION SYSTEMS AND WHAT CAN BE DONE TO REDUCE THEM. , 2000 .

[97]  Maneesh Agrawala,et al.  Zone and polygon menus: using relative position to increase the breadth of multi-stroke marking menus , 2006, CHI.

[98]  Digby Elliott,et al.  Rescaling an acquired discrete aiming movement: Specific or general motor learning? , 1997 .

[99]  Y C Liu,et al.  Comparative study of the effects of auditory, visual and multimodality displays on drivers' performance in advanced traveller information systems , 2001, Ergonomics.

[100]  Mark Asbridge,et al.  A meta-analysis of the effects of texting on driving. , 2014, Accident; analysis and prevention.

[101]  Stephen A. Brewster,et al.  Novel Multimodal Feedback Techniques for In-Car Mid-Air Gesture Interaction , 2017, AutomotiveUI.

[102]  E. R. Crossman,et al.  Feedback Control of Hand-Movement and Fitts' Law , 1983, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[103]  Charles S. Watson,et al.  Factors in the design of effective auditory displays , 1994 .

[104]  R. Tibshirani,et al.  Association between cellular-telephone calls and motor vehicle collisions. , 1997, The New England journal of medicine.

[105]  R. Hyman.  Stimulus information as a determinant of reaction time. , 1953, Journal of experimental psychology.

[106]  Eric Rodgman,et al.  The role of driver distraction in traffic crashes , 2001 .

[107]  I.,et al.  Fitts' Law as a Research and Design Tool in Human-Computer Interaction , 1992, Hum. Comput. Interact..

[108]  Roope Raisamo,et al.  Non-visual feedback cues for pen computing , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[109]  D. Elliott,et al.  Asymmetries in the regulation of visually guided aiming. , 1993, Journal of motor behavior.

[110]  S. Gelfand.  Essentials of Audiology , 1997 .

[111]  Thorsten Joachims,et al.  Eye-tracking analysis of user behavior in WWW search , 2004, SIGIR '04.

[112]  H. Branch Coslett,et al.  Two-component models of reaching: Evidence from deafferentation in a Fitts’ law task , 2009, Neuroscience Letters.

[113]  Alfred O. Effenberg.  MULTIMODAL CONVERGENT INFORMATION ENHANCES PERCEPTION ACCURACY OF HUMAN MOVEMENT PATTERNS , 2001 .

[114]  Elizabeth D. Mynatt,et al.  The Mercator Environment: A Nonvisual Interface to X Windows and Unix Workstations , 1992 .

[115]  Myounghoon Jeon,et al.  Menu Navigation With In-Vehicle Technologies: Auditory Menu Cues Improve Dual Task Performance, Preference, and Workload , 2015, Int. J. Hum. Comput. Interact..

[116]  N. Cairnie,et al.  Using finger-pointing to operate secondary controls in automobiles , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[117]  Denis Mottet,et al.  The dynamics of goal-directed rhythmical aiming , 1999, Biological Cybernetics.

[118]  Gregory Kramer,et al.  An introduction to auditory display , 1994 .

[119]  David L. Strayer,et al.  Driven to Distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular Telephone , 2001, Psychological science.

[120]  C. MacKenzie,et al.  Three-Dimensional Movement Trajectories in Fitts' Task: Implications for Control , 1987 .

[121]  D. Elliott,et al.  The Influence of Premovement Visual Information on Manual Aiming , 1987, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[122]  Michael A. Nees,et al.  Memory for Auditory Icons and Earcons with Localization Cues , 2007 .

[123]  Colin G. Drury,et al.  Application of Fitts' Law to Foot-Pedal Design , 1975 .

[124]  David J. Wheatley,et al.  Using Driver Performance Measures to Estimate Workload , 2002 .

[125]  Gregory Kramer,et al.  Auditory Display: Sonification, Audification, And Auditory Interfaces , 1994 .

[126]  Stephen A. Brewster,et al.  A Study on Gestural Interaction with a 3D Audio Display , 2004, Mobile HCI.

[127]  H. Zelaznik,et al.  Sources of Inaccuracy in Rapid Movement , 1978 .

[128]  Frank Drews,et al.  Profiles in Driver Distraction: Effects of Cell Phone Conversations on Younger and Older Drivers , 2004, Hum. Factors.

[129]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[130]  D. Sugden.  Movement speed in children. , 1980, Journal of motor behavior.

[131]  Mohan M. Trivedi,et al.  Hand gesture-based visual user interface for infotainment , 2012, AutomotiveUI.

[132]  Pierre Dragicevic,et al.  Earpod: eyes-free menu selection using touch input and reactive audio feedback , 2007, CHI.

[133]  J. W. Chaffee.  Methods for Determining Driver Reach Capability , 1969 .

[134]  Lorraine G. Kisselburgh,et al.  Rapid visual feedback processing in single-aiming movements. , 1983, Journal of motor behavior.

[135]  Micah Alpern,et al.  Developing a car gesture interface for use as a secondary task , 2003, CHI Extended Abstracts.

[136]  Richard J. Hanowski,et al.  Driver Distraction in Commercial Vehicle Operations , 2009 .

[137]  Carl Gutwin,et al.  Air pointing: Design and evaluation of spatial target acquisition with and without visual feedback , 2011, Int. J. Hum. Comput. Stud..

[138]  Romeo Chua,et al.  Discrete vs. continuous visual control of manual aiming , 1991 .

[139]  Fergus R. Fricke,et al.  Duration of Perceived and Performed Sounds , 1997 .

[140]  Umberto Castiello,et al.  Comparing Effects of 2-D and 3-D Visual Cues During Aurally Aided Target Acquisition , 2004, Hum. Factors.

[141]  Denis Mottet,et al.  Behind Fitts' law: kinematic patterns in goal-directed movements , 2004, Int. J. Hum. Comput. Stud..

[142]  Terry C Lansdown,et al.  Distraction from multiple in-vehicle secondary tasks: vehicle performance and mental workload implications , 2004, Ergonomics.

[143]  Tovi Grossman,et al.  Pointing at trivariate targets in 3D environments , 2004, CHI.