Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study

The aim of this study was (1) to provide behavioral evidence for multimodal feature integration in an object recognition task in humans and (2) to characterize the processing stages and the neural structures where multisensory interactions take place. Event-related potentials (ERPs) were recorded from 30 scalp electrodes while subjects performed a forced-choice reaction-time categorization task: At each trial, the subjects had to indicate which of two objects was presented by pressing one of two keys. The two objects were defined by auditory features alone, visual features alone, or the combination of auditory and visual features. Subjects were more accurate and rapid at identifying multimodal than unimodal objects. Spatiotemporal analysis of ERPs and scalp current densities revealed several auditory-visual interaction components temporally, spatially, and functionally distinct before 200 msec poststimulus. The effects observed were (1) in visual areas, new neural activities (as early as 40 msec poststimulus) and modulation (amplitude decrease) of the N185 wave to unimodal visual stimulus, (2) in the auditory cortex, modulation (amplitude increase) of subcomponents of the unimodal auditory N1 wave around 90 to 110 msec, and (3) new neural activity over the right fronto-temporal area (140 to 165 msec). Furthermore, when the subjects were separated into two groups according to their dominant modality to perform the task in unimodal conditions (shortest reaction time criteria), the integration effects were found to be similar for the two groups over the nonspecific fronto-temporal areas, but they clearly differed in the sensory-specific cortices, affecting predominantly the sensory areas of the nondominant modality. Taken together, the results indicate that multisensory integration is mediated by flexible, highly adaptive physiological processes that can take place very early in the sensory processing chain and operate in both sensory-specific and nonspecific cortical structures in different ways.

[1]  R. Held.  Shifts in binaural localization after prolonged exposures to atypical combinations of stimuli. , 1955, The American journal of psychology.

[2]  M HERSHENSON,et al.  Reaction time as a measure of intersensory facilitation. , 1962, Journal of experimental psychology.

[3]  W. Walter.  THE CONVERGENCE AND INTERACTION OF VISUAL, AUDITORY, AND TACTILE RESPONSES IN HUMAN NONSPECIFIC CORTEX , 1964, Annals of the New York Academy of Sciences.

[4]  D. H. Warren,et al.  Sensory conflict in judgments of spatial direction , 1969 .

[5]  W. Ritter,et al.  The sources of auditory evoked responses recorded from the human scalp. , 1970, Electroencephalography and clinical neurophysiology.

[6]  I H Bernstein,et al.  Can we see and hear at the same time? Some recent studies of intersensory facilitation of reaction time. , 1970, Acta psychologica.

[7]  T. Powell,et al.  An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. , 1970, Brain : a journal of neurology.

[8]  P A Osterhammel,et al.  Slow vertex potentials: interactions among auditory, tactile, electric and visual stimuli. , 1972, Electroencephalography and clinical neurophysiology.

[9]  FRANK MORRELL,et al.  Visual System's View of Acoustic Space , 1972, Nature.

[10]  B. Gordon,et al.  Receptive fields in deep layers of cat superior colliculus. , 1973, Journal of neurophysiology.

[11]  C. R. Michael,et al.  Integration of auditory information in the cat's visual cortex. , 1973, Vision research.

[12]  J. Andreassi,et al.  Effects of bisensory stimulation on reaction time and the evoked cortical potential , 1975 .

[13]  J. Wolpaw,et al.  A temporal component of the auditory evoked response. , 1975, Electroencephalography and clinical neurophysiology.

[14]  M. Posner,et al.  Visual dominance: an information-processing account of its origins and significance. , 1976, Psychological review.

[15]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[16]  L. Benevento,et al.  Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey , 1977, Experimental Neurology.

[17]  B. R. Shelton,et al.  The influence of vision on the absolute identification of sound-source position , 1980, Perception & psychophysics.

[18]  P. Bertelson,et al.  Cross-modal bias and perceptual fusion with auditory-visual spatial discordance , 1981, Perception & psychophysics.

[19]  Leslie G. Ungerleider.  Two cortical visual systems , 1982 .

[20]  Jeff Miller,et al.  Divided attention: Evidence for coactivation with redundant signals , 1982, Cognitive Psychology.

[21]  S. Ito.  Prefrontal unit activity of macaque monkeys during auditory and visual reaction time tasks , 1982, Brain Research.

[22]  B. Stein,et al.  Interactions among converging sensory inputs in the superior colliculus. , 1983, Science.

[23]  D. H. Warren,et al.  Discrepancy and nondiscrepancy methods of assessing visual-auditory interaction , 1983, Perception & psychophysics.

[24]  David L. Sparks,et al.  Auditory receptive fields in primate superior colliculus shift with changes in eye position , 1984, Nature.

[25]  R. Schmidt,et al.  The locus of intersensory facilitation of reaction time. , 1984, Acta psychologica.

[26]  O. Fehér,et al.  Dynamic interactions of evoked potentials in a polysensory cortex of the cat , 1984, Neuroscience.

[27]  Gillian Rhodes,et al.  Cross-modal effects on visual and auditory object perception , 1984, Perception & psychophysics.

[28]  B. Stein,et al.  Spatial factors determine the activity of multisensory neurons in cat superior colliculus , 1986, Brain Research.

[29]  E. Vaadia,et al.  Unit study of monkey frontal cortex: active localization of auditory and of visual stimuli. , 1986, Journal of neurophysiology.

[30]  B. Stein,et al.  Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. , 1986, Journal of neurophysiology.

[31]  Jeff Miller,et al.  Timecourse of coactivation in bimodal divided attention , 1986, Perception & psychophysics.

[32]  M. Scherg,et al.  Evoked dipole source potentials of the human auditory cortex. , 1986, Electroencephalography and clinical neurophysiology.

[33]  C. K. Peck.  Visual-auditory interactions in cat superior colliculus: their role in the control of gaze , 1987, Brain Research.

[34]  Keiji Tanaka,et al.  Polysensory properties of neurons in the anterior bank of the caudal superior temporal sulcus of the macaque monkey. , 1988, Journal of neurophysiology.

[35]  M. Alex Meredith,et al.  Neurons and behavior: the same rules of multisensory integration apply , 1988, Brain Research.

[36]  Lawrence G. McDade,et al.  Behavioral Indices of Multisensory Integration: Orientation to Visual Cues is Affected by Auditory Stimuli , 1989, Journal of Cognitive Neuroscience.

[37]  F. Perrin,et al.  Spherical splines for scalp potential and current density mapping. , 1989, Electroencephalography and clinical neurophysiology.

[38]  Crossmodal Interactions of Auditory Stimulus Presentation on the Visual Evoked Magnetic Response , 1989 .

[39]  C. Eriksen,et al.  Coactivation in the perception of redundant targets. , 1990, Journal of Experimental Psychology: Human Perception and Performance.

[40]  S J Luck,et al.  Visual event-related potentials index focused attention within bilateral stimulus arrays. II. Functional dissociation of P1 and N1 components. , 1990, Electroencephalography and clinical neurophysiology.

[41]  R. Hari,et al.  Seeing speech: visual information from lip movements modifies activity in the human auditory cortex , 1991, Neuroscience Letters.

[42]  Eiichi Iwai,et al.  Neuronal activity in visual, auditory and polysensory areas in the monkey temporal cortex during visual fixation task , 1991, Brain Research Bulletin.

[43]  D. Gaffan,et al.  Auditory-visual associations, hemispheric specialization and temporal-frontal interaction in the rhesus monkey. , 1991, Brain : a journal of neurology.

[44]  J Pernier,et al.  Computer-assisted placement of electrodes on the human head. , 1992, Electroencephalography and clinical neurophysiology.

[45]  Yasuhiko Tamai,et al.  Sensory response of cortical neurons in the anterior ectosylvian sulcus, including the area evoking eye movement , 1992, Brain Research.

[46]  M. Wallace,et al.  Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. , 1993, Journal of neurophysiology.

[47]  S. Hillyard,et al.  Identification of early visual evoked potential generators by retinotopic and topographic analyses , 1994 .

[48]  P. Reuter-Lorenz,et al.  Visual-auditory interactions in sensorimotor processing: saccades versus manual responses. , 1994, Journal of experimental psychology. Human perception and performance.

[49]  M. Gazzaniga,et al.  Combined spatial and temporal imaging of brain activity during visual selective attention in humans , 1994, Nature.

[50]  The cognitive architecture of bimodal event perception: A commentary and addendum to Radeau (1994) , 1994 .

[51]  Mark T. Wallace,et al.  Development and neural basis of multisensory integration. , 1994 .

[52]  Michael D. Rugg,et al.  Word and Nonword Repetition Within- and Across-Modality: An Event-Related Potential Study , 1995, Journal of Cognitive Neuroscience.

[53]  D. Barth,et al.  The spatiotemporal organization of auditory, visual, and auditory-visual evoked potentials in rat cortex , 1995, Brain Research.

[54]  S Baron-Cohen,et al.  The physiology of coloured hearing. A PET activation study of colour-word synaesthesia. , 1995, Brain : a journal of neurology.

[55]  W Singer,et al.  Visual feature integration and the temporal correlation hypothesis. , 1995, Annual review of neuroscience.

[56]  M. Frens,et al.  Spatial and temporal factors determine auditory-visual interactions in human saccadic eye movements , 1995, Perception & psychophysics.

[57]  G. Mangun.  Neural mechanisms of visual selective attention. , 1995, Psychophysiology.

[58]  A. Treisman.  The binding problem , 1996, Current Opinion in Neurobiology.

[59]  V. Jousmäki,et al.  Temporal integration in auditory sensory memory: neuromagnetic evidence. , 1996, Electroencephalography and clinical neurophysiology.

[60]  S. Hillyard,et al.  Selective attention to the color and direction of moving stimuli: Electrophysiological correlates of hierarchical feature selection , 1996, Perception & psychophysics.

[61]  S. Hillyard,et al.  Spatial Selective Attention Affects Early Extrastriate But Not Striate Components of the Visual Evoked Potential , 1996, Journal of Cognitive Neuroscience.

[62]  T L Hubbard,et al.  Synesthesia-like mappings of lightness, pitch, and melodic interval. , 1996, The American journal of psychology.

[63]  W H Ehrenstein,et al.  A Cross-Modal Aftereffect: Auditory Displacement following Adaptation to Visual Motion , 1996, Perceptual and motor skills.

[64]  D. Perrett,et al.  Integration of form and motion in the anterior superior temporal polysensory area (STPa) of the macaque monkey. , 1996, Journal of neurophysiology.

[65]  V. Walsh,et al.  Perception: The seeing ear , 1996, Current Biology.

[66]  M. Wallace,et al.  Representation and integration of multiple sensory inputs in primate superior colliculus. , 1996, Journal of neurophysiology.

[67]  Denis Fize,et al.  Speed of processing in the human visual system , 1996, Nature.

[68]  M T Wallace,et al.  Comparisons of cross-modality integration in midbrain and cortex. , 1996, Progress in brain research.

[69]  B. Stein,et al.  Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis , 1996, Journal of Cognitive Neuroscience.

[70]  R. Sekuler,et al.  Sound alters visual motion perception , 1997, Nature.

[71]  E. Schröger.  On the detection of auditory deviations: a pre-attentive activation model. , 1997, Psychophysiology.

[72]  J. Driver,et al.  Audiovisual links in exogenous covert spatial orienting , 1997, Perception & psychophysics.

[73]  E. Schröger,et al.  Speeded responses to audiovisual signal changes result from bimodal integration. , 1998, Psychophysiology.

[74]  C. Spence,et al.  Multisensory perception: Beyond modularity and convergence , 2000, Current Biology.

[75]  B. Rossion,et al.  The time‐course of intermodal binding between seeing and hearing affective information , 2000, Neuroreport.

[76]  Paul J. Laurienti,et al.  Neural mechanisms for integrating information from multiple senses , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[77]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[78]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[79]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.