Multisensory aspects of the spatial cuing of driver attention

 The primary goal of the empirical research outlined in this thesis was to examine a number of the factors contributing to the design of more ergonomic multisensory warning signals, that is, signals that elicit efficient and effective responses from interface operators under demanding conditions. To achieve this goal, a series of experiments was conducted in order to examine the nature and consequences of the crossmodal links inherent in spatial attention between various different sensory modalities, such as audition, vision, and touch, in an applied setting. In particular, a laboratory-based simulated driving task was used to investigate the effectiveness of various different auditory and vibrotactile cues in orienting a driver's attention to potential emergency driving events seen through the front windscreen or rearview mirror. The results of the first set of auditory spatial cuing experiments highlighted a significant performance advantage when the target driving events occurred in the cued, rather than the uncued, direction, with the biggest benefits being seen following spatially-predictive auditory or verbal cues. The second set of vibrotactile spatial cuing experiments demonstrated the potential utility of vibrotactile warning signals for presenting spatial information to car drivers, and the results were successfully replicated in a study using a high-fidelity driving simulator. The third set of experiments, incorporating an orthogonal task design, were conducted in order to examine the mechanisms responsible for the advantageous spatial cuing effects reported in the earlier experiments. Taken together, the results demonstrated that while directional congruency between a warning signal and target driving event may be sufficient to facilitate performance due to the priming of the appropriate response, attentional facilitation (i.e., perceptual enhancement) typically requires the co-location of the cue and target within the same functional region of space. In sum, this thesis demonstrates the potential value of approaching the design of effective multisensory warning signals for human operators by studying the information processing mechanisms in the human brain. The findings outlined here add to the literature concerning the brain's differential representation of stimuli presented in peripersonal as opposed to extrapersonal space. Further experimental chapters detail experiments that examined verbal directional cuing, olfactory cuing, and crossmodal interactions in virtual haptic environments.

[1]  N. Durlach,et al.  Manual discrimination of force using active finger motion , 1991, Perception & psychophysics.

[2]  T Z Strybel,et al.  Aurally Aided Visual Search in the Central Visual Field: Effects of Visual Load and Visual Enhancement of the Target , 1991, Human factors.

[3]  Ronald A. Rensink,et al.  Change-blindness as a result of ‘mudsplashes’ , 1999, Nature.

[4]  Lynn A. Streeter,et al.  How to Tell People Where to Go: Comparing Navigational Aids , 1985, Int. J. Man Mach. Stud..

[5]  Rob H. J. van der Lubbe,et al.  Interruption from irrelevant auditory and visual onsets even when attention is in a focused state , 2005, Experimental Brain Research.

[6]  Cristy Ho,et al.  Verbal interface design: Do verbal directional cues automatically orient visual spatial attention? , 2006, Comput. Hum. Behav..

[7]  S. Vecera,et al.  What are you looking at? Impaired ‘social attention’ following frontal-lobe damage , 2004, Neuropsychologia.

[8]  Ryan M. Traylor,et al.  A Haptic Back Display for Attentional and Directional Cueing , 2003 .

[9]  M. Spapé,et al.  Waypoint navigation on land : different ways of coding distance to the next waypoint , 2004 .

[10]  H. Levitt Transformed up-down methods in psychoacoustics. , 1971, The Journal of the Acoustical Society of America.

[11]  M. Turatto,et al.  Attentional capture by color without any relevant attentional set , 2001, Perception & psychophysics.

[12]  H Pashler,et al.  Shifting visual attention and selecting motor responses: distinct attentional mechanisms. , 1991, Journal of experimental psychology. Human perception and performance.

[13]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[14]  K Rumar,et al.  The basic driver error: late detection. , 1990, Ergonomics.

[15]  J. Stroop Studies of interference in serial verbal reactions. , 1992 .

[16]  H S Vitense,et al.  Multimodal feedback: an assessment of performance and mental workload , 2003, Ergonomics.

[17]  R. Morris,et al.  Response Times to Visual and Auditory Alarms during Anaesthesia , 1996, Anaesthesia and intensive care.

[18]  Barbara G. Shinn-Cunningham,et al.  Selective and Divided Attention: Extracting Information from Simultaneous Sound Sources , 2004, ICAD.

[19]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[20]  M. Ernst,et al.  Integration of force and position cues for shape perception through active touch , 2006, Brain Research.

[21]  J.B.F. van Erp,et al.  A multi-purpose tactile vest for astronauts in the international space station , 2003 .

[22]  Phillip R. Zoladz,et al.  Cognitive Enhancement through Stimulation of the Chemical Senses , 2005 .

[23]  Steven Peterson,et al.  Are Drivers of Air-Bag-Equipped Cars More Aggressive? A Test of the Offsetting Behavior Hypothesis , 1995, Journal law and economy.

[24]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[25]  C. A. Simpson,et al.  Synthesized Speech Rate and Pitch Effects on Intelligibility of Warning Messages for Pilots , 1984, Human factors.

[26]  Hong Z. Tan,et al.  Development of a wearable haptic display for situation awareness in altered-gravity environment: some initial findings , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[27]  Bryan Raudenbush Enhancing athletic performance through the administration of peppermint odor , 2001 .

[28]  Karen M. Arnell,et al.  Attention to visual pattern information produces the attentional blink in rapid serial visual presentation. , 1994, Journal of experimental psychology. Human perception and performance.

[29]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[30]  Albert Kircher,et al.  Using mobile telephones: cognitive workload and attention resource allocation. , 2004, Accident; analysis and prevention.

[31]  S. Gepshtein,et al.  The combination of vision and touch depends on spatial proximity. , 2005, Journal of vision.

[32]  D. Strayer,et al.  Cell phone-induced failures of visual attention during simulated driving. , 2003, Journal of experimental psychology. Applied.

[33]  Hermann Ackermann,et al.  Spatial auditory attention is modulated by tactile priming , 2005, Experimental Brain Research.

[34]  Gregory O. Gibson,et al.  Tactile spatial sensitivity and anisotropy , 2005, Perception & psychophysics.

[35]  R. Nelson,et al.  Reaction times for hand movements made in response to visual versus vibratory cues. , 1990, Somatosensory & motor research.

[36]  C. Spence,et al.  Assessing the effectiveness of various auditory cues in capturing a driver's visual attention. , 2005, Journal of experimental psychology. Applied.

[37]  Sharon L. Oviatt,et al.  When do we interact multimodally?: cognitive load and multimodal communication patterns , 2004, ICMI '04.

[38]  Angus H. Rupert,et al.  The use of a tactile interface to convey position and motion perceptions , 1994 .

[39]  W. Brouwer,et al.  Age differences in divided attention in a simulated driving task. , 1988, Journal of gerontology.

[40]  David L. Smith,et al.  Analysis of Braking and Steering Performance in Car- Following Scenarios , 2003 .

[41]  Charles Spence,et al.  On the inability to ignore touch when responding to vision in the crossmodal congruency task. , 2005, Acta psychologica.

[42]  T. Seizova-Cajic,et al.  Size perception by vision and kinesthesia , 1998, Perception & psychophysics.

[43]  Mark T. Wallace,et al.  Crossmodal spatial interactions in subcortical and cortical circuits , 2004 .

[44]  Philip H. K. Seymour,et al.  Stroop interference in naming and verifying spatial locations , 1973 .

[45]  H. J. Muller,et al.  Reflexive and voluntary orienting of visual attention: time course of activation and resistance to interruption. , 1989, Journal of experimental psychology. Human perception and performance.

[46]  Vincent Hayward,et al.  Force can overcome object geometry in the perception of shape through active touch , 2001, Nature.

[47]  S S Hsiao,et al.  Detection of vibration transmitted through an object grasped in the hand. , 1999, Journal of neurophysiology.

[48]  J. Williams,et al.  Haptic chameleon: a new concept of shape-changing user interface controls with force feedback , 2004, CHI EA '04.

[49]  A. Muzet,et al.  Effect of driving duration and partial sleep deprivation on subsequent alertness and performance of car drivers , 2005, Physiology & Behavior.

[50]  R. Klatzky,et al.  There's more to touch than meets the eye: The salience of object attributes for haptics with and without vision. , 1987 .

[51]  J Godthelp,et al.  EXPLORATORY SIMULATOR STUDY ON THE USE OF ACTIVE CONTROL DEVICES IN CAR DRIVING , 1992 .

[52]  J. Royet,et al.  Lateralization of olfactory processes. , 2004, Chemical senses.

[53]  Michel Guerraz,et al.  Reference frames and haptic perception of orientation: Body and head tilt effects on the oblique effect , 2001, Perception & psychophysics.

[54]  Charles Spence,et al.  Olfactory facilitation of dual-task performance , 2005, Neuroscience Letters.

[55]  Sharon Oviatt,et al.  Multimodal Interfaces , 2008, Encyclopedia of Multimedia.

[56]  J. Findlay,et al.  The Relationship between Eye Movements and Spatial Attention , 1986, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[57]  D. H. Warren,et al.  Sensory conflict in judgments of spatial direction , 1969 .

[58]  C. Spence,et al.  A crossmodal attentional blink between vision and touch , 2002, Psychonomic bulletin & review.

[59]  William Prinzmetal,et al.  Attention: reaction time and accuracy reveal different mechanisms. , 2005, Journal of experimental psychology. General.

[60]  B. Pause,et al.  Chemosensory anxiety signals augment the startle reflex in humans , 2006, Neuroscience Letters.

[61]  D J Newman,et al.  A tactile display for international space station (ISS) extravehicular activity (EVA). , 2000, Aviation, space, and environmental medicine.

[62]  S. Monsell,et al.  Costs of a predictible switch between simple cognitive tasks. , 1995 .

[63]  Keith Wesnes,et al.  AROMAS OF ROSEMARY AND LAVENDER ESSENTIAL OILS DIFFERENTIALLY AFFECT COGNITION AND MOOD IN HEALTHY ADULTS , 2003, The International journal of neuroscience.

[64]  Richard L. McKinley,et al.  Aurally Aided Visual Search under Virtual and Free-Field Listening Conditions , 1996, Hum. Factors.

[65]  Louis Tijerina,et al.  PRELIMINARY STUDIES IN HAPTIC DISPLAYS FOR REAR-END COLLISION AVOIDANCE SYSTEM AND ADAPTIVE CRUISE CONTROL SYSTEM APPLICATIONS , 2000 .

[66]  C Spence,et al.  Shared Attentional Resources for processing Visual and Chemosensory Information , 2001, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[67]  Nadine B. Sarter,et al.  Good Vibrations: Tactile Feedback in Support of Attention Allocation and Human-Automation Coordination in Event-Driven Domains , 1999, Hum. Factors.

[68]  John J. Foxe,et al.  Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. , 2005, Cerebral cortex.

[69]  Jon Driver,et al.  Covert Spatial Orienting in Audition: Exogenous and Endogenous Mechanisms , 1994 .

[70]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[71]  J. Pratt,et al.  Inhibition of return in location- and identity-based choice decision tasks , 1997, Perception & psychophysics.

[72]  J R Treat,et al.  TRI-LEVEL STUDY OF THE CAUSES OF TRAFFIC ACCIDENTS: FINAL REPORT , 1979 .

[73]  Ronald E. Shor,et al.  The processing of conceptual information on spatial directions from pictorial and linguistic symbols , 1970 .

[74]  R. Yerkes,et al.  The relation of strength of stimulus to rapidity of habit‐formation , 1908 .

[75]  C. Spence,et al.  Audiotactile temporal order judgments. , 2005, Acta psychologica.

[76]  D. A. Stevens,et al.  Individual differences in the perceived intensity and quality of specific odors following self- and cross-adaptation. , 1994, Chemical senses.

[77]  Stephen J. Boies,et al.  Components of attention. , 1971 .

[78]  Edward A. Essock,et al.  Superior sensitivity for tactile stimuli oriented proximally-distally on the finger : Implications for mixed class 1 and class 2 anisotropies , 1997 .

[79]  L. M. Warner,et al.  The Neural Mechanisms for Minimizing Cross-Modal Distraction , 2004, The Journal of Neuroscience.

[80]  Motor vehicle accidents and vehicular pollution in Los Angeles. , 1972, Archives of environmental health.

[81]  C. Spence,et al.  Cross-modal links in exogenous covert spatial orienting between touch, audition, and vision , 1998, Perception & psychophysics.

[82]  G. Mowbray,et al.  MAN'S SENSES AS INFORMATIONAL CHANNELS, , 1958 .

[83]  Benjamin White,et al.  Perceptual Findings with the Vision-Substitution System , 1970 .

[84]  D. Bavelier,et al.  Components of visual prior entry , 2003, Cognitive Psychology.

[85]  Rupert Ah Tactile situation awareness system: proprioceptive prostheses for sensory deficiencies. , 2000 .

[86]  C. Spence,et al.  Crossmodal links between vision and touch in covert endogenous spatial attention. , 2000, Journal of experimental psychology. Human perception and performance.

[87]  G. Tassinari,et al.  Consequences of covert orienting to non-informative stimuli of different modalities: A unitary mechanism? , 1996, Neuropsychologia.

[88]  James T. Townsend,et al.  The Stochastic Modeling of Elementary Psychological Processes , 1983 .

[89]  Alan Kingstone,et al.  On audiovisual spatial synergy: The fragility of the phenomenon , 2005, Perception & psychophysics.

[90]  C. Spence,et al.  Beyond the body schema: Visual, prosthetic, and technological contributions to bodily perception and awareness , 2006 .

[91]  Ronald A. Rensink,et al.  Change blindness: past, present, and future , 2005, Trends in Cognitive Sciences.

[92]  K. Saberi,et al.  Auditory psychomotor coordination and visual search performance , 1990, Perception & psychophysics.

[93]  J. Werker,et al.  Tuned to the signal: the privileged status of speech for young infants. , 2004, Developmental science.

[94]  Boris M. Velichkovsky,et al.  Change detection and occlusion modes in road-traffic scenarios , 2002 .

[95]  David L. Strayer,et al.  Driven to Distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular Telephone , 2001, Psychological science.

[96]  G Rizzolatti,et al.  The Space Around Us , 1997, Science.

[97]  F.H. Previc,et al.  Neuropsychological guidelines for aircraft control stations , 2000, IEEE Engineering in Medicine and Biology Magazine.

[98]  W B Verwey,et al.  On-line driver workload estimation. Effects of road situation and age on secondary task measures , 2000, Ergonomics.

[99]  C. Spence,et al.  Speech Shadowing While Driving , 2003, Psychological science.

[100]  Matthew P. Reed,et al.  Comparison of driving performance on-road and in a low-cost simulator using a concurrent telephone dialling task , 1999 .

[101]  Jean-Louis Millot,et al.  Effects of ambient odors on reaction time in humans , 2002, Neuroscience Letters.

[102]  J. Findlay,et al.  Sensitivity and criterion effects in the spatial cuing of visual attention , 1987, Perception & psychophysics.

[103]  Takamasa Suetomi,et al.  DRIVER BEHAVIOR UNDER A COLLISION WARNING SYSTEM : A DRIVING SIMULATOR STUDY , 1997 .

[104]  Frank P. McKenna,et al.  Integrating multiple information sources: using redundancy in the design of warnings , 1995 .

[105]  Cristy Ho,et al.  Assessing the effectiveness of "intuitive" vibrotactile warning signals in preventing front-to-rear-end collisions in a driving simulator. , 2006, Accident; analysis and prevention.

[106]  E. B. Newman,et al.  The localization of actual sources of sound. , 1936 .

[107]  H E Egeth,et al.  Response time and accuracy revisited: converging support for the interactive race model. , 1993, Journal of experimental psychology. Human perception and performance.

[108]  T H Rockwell,et al.  Spare Visual Capacity in Driving-Revisited: New Empirical Results for an Old Idea , 1988 .

[109]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[110]  Walter W. Wierwille,et al.  The Influence of Motion and Audio Cues on Driver Performance in an Automobile Simulator , 1975 .

[111]  J. V. Erp,et al.  Vibrotactile in-vehicle navigation system , 2004 .

[112]  Immanuel Barshi,et al.  Effects of instruction modality and readback on accuracy in following navigation commands. , 2004, Journal of experimental psychology. Applied.

[114]  Cristy Ho Using spatial warning signals to capture a driver's visual attention , 2004, ICMI '04.

[115]  K. Zilles,et al.  Neural consequences of acting in near versus far space: a physiological basis for clinical dissociations. , 2000, Brain : a journal of neurology.

[116]  R. Shiffrin,et al.  Controlled and automatic human information processing: I , 1977 .

[117]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[118]  G. A. Miller,et al.  The intelligibility of speech as a function of the context of the test materials. , 1951, Journal of experimental psychology.

[119]  Nicholas P. Holmes,et al.  Visual bias of unseen hand position with a mirror: spatial and temporal factors , 2005, Experimental Brain Research.

[120]  Alan Kingstone,et al.  The ventriloquist in motion: illusory capture of dynamic information across sensory modalities. , 2002, Brain research. Cognitive brain research.

[121]  Jan B. F. van Erp,et al.  Tactile Navigation Display , 2000, Haptic Human-Computer Interaction.

[122]  V S Ramachandran,et al.  Mirror agnosia , 1997, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[123]  K L Shapiro,et al.  Temporary suppression of visual processing in an RSVP task: an attentional blink? . , 1992, Journal of experimental psychology. Human perception and performance.

[124]  Josef Schumann,et al.  ON THE EFFECTIVENESS OF AN ACTIVE STEERING WHEEL IN CRITICAL DRIVING SITUATIONS - A PROVING GROUND EXPERIMENT , 1992 .

[125]  J Driver,et al.  Cross-modal selective attention: On the difficulty of ignoring sounds at the locus of visual attention , 2000, Perception & psychophysics.

[126]  W van Winsum,et al.  Time Headway in Car following and Operational Performance during Unexpected Braking , 1997, Perceptual and motor skills.

[127]  T A Mondor,et al.  Facultative and inhibitory effects of location and frequency cues: Evidence of a modulation in perceptual sensitivity , 1999, Perception & psychophysics.

[128]  Jean-Louis Millot,et al.  Ambient odors modulate visual attentional capture , 2003, Neuroscience Letters.

[129]  P. Rodway The modality shift effect and the effectiveness of warning signals in different modalities. , 2005, Acta psychologica.

[130]  A Kemeny,et al.  Perception and reproduction of force direction in the horizontal plane. , 2003, Journal of neurophysiology.

[131]  J. Gibson,et al.  Adaptation, after-effect and contrast in the perception of curved lines. , 1933 .

[132]  Gerd Kobal,et al.  Selective attention to the chemosensory modality , 2000, Perception & psychophysics.

[133]  A H Rupert An instrumentation solution for reducing spatial disorientation mishaps. , 2000, IEEE engineering in medicine and biology magazine : the quarterly magazine of the Engineering in Medicine & Biology Society.

[134]  L. A. Jones Matching Forces: Constant Errors and Differential Thresholds , 1989, Perception.

[135]  S. Charles,et al.  10 - Crossmodal Attentional Capture: A Controversy Resolved? , 2001 .

[136]  P. Calabresi,et al.  Saccade preparation inhibits reorienting to recently attended locations. , 1989, Journal of experimental psychology. Human perception and performance.

[137]  C. Spence,et al.  Multisensory Integration: Maintaining the Perception of Synchrony , 2003, Current Biology.

[138]  Jennifer M. Glass,et al.  Virtually Perfect Time Sharing in Dual-Task Performance: Uncorking the Central Cognitive Bottleneck , 2001, Psychological science.

[139]  Jon Driver,et al.  Cross-modal links in attention between audition, vision, and touch: Implications for interface design , 1997 .

[140]  Geraint Rees,et al.  What can functional imaging reveal about the role of attention in visual awareness? , 2001, Neuropsychologia.

[141]  S. M. Luria,et al.  Conflicting visual and tactual-kinesthetic stimulation , 1970 .

[142]  D. Wolpert,et al.  When Feeling Is More Important Than Seeing in Sensorimotor Adaptation , 2002, Current Biology.

[143]  W. R. Garner,et al.  The Stimulus in Information Processing , 1970 .

[144]  J. Tipples Eye gaze is not unique: Automatic orienting in response to uninformative arrows , 2002, Psychonomic bulletin & review.

[145]  A. Kingstone,et al.  Are eyes special? It depends on how you look at it , 2002, Psychonomic bulletin & review.

[146]  Christopher D. Wickens,et al.  The Structure of Attentional Resources , 1980 .

[147]  F B Simmons,et al.  High-Powered Automobile Stereos , 1993, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[148]  C. Braun,et al.  Emotional qualities of odors and their influence on the startle reflex in humans. , 1994, Psychophysiology.

[149]  M. Posner,et al.  Visual dominance: an information-processing account of its origins and significance. , 1976, Psychological review.

[150]  J. V. Van Erp,et al.  Presenting directions with a vibrotactile torso display , 2005, Ergonomics.

[151]  S. Lederman,et al.  Perception of texture by vision and touch: multidimensionality and intersensory integration. , 1986, Journal of experimental psychology. Human perception and performance.

[152]  William S. Harwin,et al.  Interaction of Visual and Haptic Information in Simulated Environments: Texture Perception , 2000, Haptic Human-Computer Interaction.

[153]  A. Senju,et al.  Direct gaze captures visuospatial attention , 2005 .

[154]  C. Spence,et al.  The cost of expecting events in the wrong sensory modality , 2001, Perception & psychophysics.

[155]  David R Moore,et al.  Auditory perception: The near and far of sound localization , 1999, Current Biology.

[156]  I. THE ATTENTION SYSTEM OF THE HUMAN BRAIN , 2002 .

[157]  R. Brent Gillespie,et al.  Shared Control between Human and Machine: Using a Haptic Steering Wheel to Aid in Land Vehicle Guidance , 2001 .

[158]  Joel S. Warm,et al.  Effects of olfactory stimulation on performance and stress in a visual sustained attention task , 1991 .

[159]  M. Posner,et al.  Components of visual orienting , 1984 .

[160]  M. Sivak The Information That Drivers Use: Is it Indeed 90% Visual? , 1996, Perception.

[161]  R. Klein Attention and visual dominance: a chronometric analysis. , 1977, Journal of experimental psychology. Human perception and performance.

[162]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[163]  Jacques Paillard,et al.  The hand and the tool: the functional architecture of human technical skills , 1993 .

[164]  T A Mondor,et al.  Effect of same- and different-modality spatial cues on auditory and visual target identification. , 1998, Journal of experimental psychology. Human perception and performance.

[165]  G. VonBékésy,et al.  INTERACTION OF PAIRED SENSORY STIMULI AND CONDUCTION IN PERIPHERAL NERVES. , 1963, Journal of applied physiology.

[166]  Jon Driver,et al.  A new approach to the design of multimodal warning signals , 1999 .

[167]  I ROCK,et al.  Vision and Touch: An Experimentally Created Conflict between the Two Senses , 1964, Science.

[168]  C. Spence,et al.  Exogenous spatial cuing studies of human crossmodal attention and multisensory integration , 2004 .