Electrophysiology of spatial scene analysis: the mismatch negativity (MMN) is sensitive to the ventriloquism illusion

OBJECTIVES The ventriloquism effect is the tendency to underestimate the spatial separation between synchronous auditory and visual signals moderately separated in space. If, as it is thought, this effect is pre-attentive, it could modulate the mismatch negativity (MMN) that indexes the automatic, pre-attentive detection of deviant auditory stimuli rarely occurring in a sequence of standard stimuli. We assessed the existence of an MMN evoked by auditory and visual signals made up of standard sounds coming from the same location as the visual signal and deviant sounds coming from lateral deviations (20 or 60 degrees). As first observed in a behavioral study, a ventriloquism effect occurred for 20 degrees spatial separation but not for 60 degrees. METHODS Cortical evoked potentials were recorded using the oddball paradigm on 8 adults in auditory alone and audiovisual conditions. RESULTS In the auditory alone condition, each spatial localization contrast elicited a significant MMN. In the audiovisual condition, a significant MMN was only evoked for the 60 degrees contrasts. CONCLUSIONS The MMN evoked by spatial separation contrasts (20 and 60 degrees) in the auditory alone condition was suppressed by the corresponding audiovisual condition only when the latter yielded a ventriloquism effect, suggesting that this effect occurs at an early perceptual level.

[1]  O. Longe,et al.  Auditory evoked potentials to abrupt pitch and timbre change of complex tones: electrophysiological evidence of 'streaming'? , 1998, Electroencephalography and clinical neurophysiology.

[2]  R. Cammann.  Is there a mismatch negativity (MMN) in visual modality? , 1990, Behavioral and Brain Sciences.

[3]  I. Winkler,et al.  Auditory stream segregation processes operate similarly in school-aged children and adults , 2001, Hearing Research.

[4]  Mark T. Wallace,et al.  Chapter 8 The visually responsive neuron and beyond: multisensory integration in cat and monkey , 1993 .

[5]  W. Ritter,et al.  Attention affects the organization of auditory input associated with the mismatch negativity system , 1998, Brain Research.

[6]  N. Cowan,et al.  The Role of Large-Scale Memory Organization in the Mismatch Negativity Event-Related Brain Potential , 2001, Journal of Cognitive Neuroscience.

[7]  Paul Bertelson,et al.  The effect of a textured visual field on modality dominance in a ventriloquism situation , 1976 .

[8]  R. Uwer,et al.  Stability of mismatch negativities in children , 2000, Clinical Neurophysiology.

[9]  William H. Press,et al.  Book-Review - Numerical Recipes in Pascal - the Art of Scientific Computing , 1989 .

[10]  D. Holender,et al.  Analytic approaches to human cognition , 1992 .

[11]  R. Näätänen,et al.  Mismatch negativity--the measure for central sound representation accuracy. , 1997, Audiology & neuro-otology.

[12]  O. Aaltonen,et al.  Practical Issues in the Clinical Application of Mismatch Negativity , 1995, Ear and hearing.

[13]  B. Stein,et al.  The Merging of the Senses , 1993 .

[14]  K. Krauel,et al.  Chemosensory event-related potentials (CSERP) as a key to the psychology of odors. , 2000, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[15]  D. Guthrie,et al.  Significance testing of difference potentials. , 1991, Psychophysiology.

[16]  P. Bertelson,et al.  Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations , 1977 .

[17]  R. Lynn,et al.  A secular decline in the strength of Spearman’s g in Japan , 1994 .

[18]  R Näätänen,et al.  Mismatch negativity--a unique measure of sensory processing in audition. , 1995, The International journal of neuroscience.

[19]  W. Ritter,et al.  An investigation of the auditory streaming effect using event-related brain potentials. , 1999, Psychophysiology.

[20]  P. Bertelson,et al.  The After-Effects of Ventriloquism , 1974, The Quarterly journal of experimental psychology.

[21]  H. Davis,et al.  Principles of electric response audiometry. , 1976, The Annals of otology, rhinology, and laryngology.

[22]  The dissection of intersensory bias: weighting for Radeau , 1994, Current psychology of cognition = Cahiers de psychologie cognitive : CPC.

[23]  C. V. Jackson,et al.  Visual Factors in Auditory Localization , 1953 .

[24]  E. Schröger,et al.  Fast preattentive processing of location: a functional basis for selective listening in humans , 1997, Neuroscience Letters.

[25]  B. Stein,et al.  Multisensory integration. Neural and behavioral solutions for dealing with stimuli from different sensory modalities. , 1990, Annals of the New York Academy of Sciences.

[26]  Robert B. Welch,et al.  The “ventriloquist effect”: Visual dominance or response bias? , 1975 .

[27]  L. K. Canon.  Intermodality inconsistency of input and directed attention as determinants of the nature of adaptation. , 1970, Journal of experimental psychology.

[28]  Object identity is not a condition but a result of intersensory integration: the case of audiovisual interactions , 2001 .

[29]  K. Reinikainen,et al.  Event-related brain potential of human newborns to pitch change of an acoustic stimulus. , 1990, Electroencephalography and clinical neurophysiology.

[30]  I. Winkler,et al.  Mismatch negativity is unaffected by top-down predictive information , 2001, Neuroreport.

[31]  N. Cowan,et al.  Two cognitive systems simultaneously prepared for opposite events. , 1999, Psychophysiology.

[32]  B. Stein,et al.  Spatial factors determine the activity of multisensory neurons in cat superior colliculus , 1986, Brain Research.

[33]  P. Bertelson,et al.  Cross-modal bias and perceptual fusion with auditory-visual spatial discordance , 1981, Perception & psychophysics.

[34]  R. Näätänen.  The role of attention in auditory information processing as revealed by event-related potentials and other brain measures of cognitive function , 1990, Behavioral and Brain Sciences.

[35]  E. Schröger.  On the detection of auditory deviations: a pre-attentive activation model. , 1997, Psychophysiology.

[36]  L. Schiebinger,et al.  Commentary on Risto Naatanen (1990). The role of attention in auditory information processing as revealed by event-related potentials and other brain measures of cognitive fenctiono BBS 13s201-2888 , 1991 .

[37]  N. Cowan.  Attention and Memory: An Integrated Framework , 1995 .

[38]  P. Leppänen,et al.  Mismatch negativity (MMN) as a tool for investigating auditory discrimination and sensory memory in infants and children , 2000, Clinical Neurophysiology.

[39]  W. Ritter,et al.  Predictability of stimulus deviance and the mismatch negativity , 1998, Neuroreport.

[40]  I. Winkler,et al.  The concept of auditory stimulus representation in cognitive neuroscience. , 1999, Psychological bulletin.

[41]  T. Carrell,et al.  Mismatch negativity event-related potential elicited by speech stimuli. , 1992, Ear and hearing.

[42]  H Hämäläinen,et al.  Is the somatosensory N250 related to deviance discrimination or conscious target detection? , 1996, Electroencephalography and clinical neurophysiology.

[43]  G. J. Thomas.  Experimental study of the influence of vision on sound localization. , 1941 .

[44]  Csaba Veres,et al.  Towards a general law of numerical/object identity. Commentary. Author's reply , 2001 .

[45]  a.R.V.,et al.  Clinical neurophysiology , 1961, Neurology.

[46]  E. Knudsen,et al.  Vision calibrates sound localization in developing barn owls , 1989, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[47]  I. Winkler,et al.  Memory prerequisites of mismatch negativity in the auditory event-related potential (ERP). , 1993, Journal of experimental psychology. Learning, memory, and cognition.

[48]  D. Javitt,et al.  The Mismatch Negativity of Event‐Related Potentials as a Probe of Transient Auditory Memory: A Review , 1995, Ear and hearing.

[49]  D. H. Warren,et al.  The role of visual-auditory “compellingness” in the ventriloquism effect: Implications for transitivity among the spatial senses , 1981, Perception & psychophysics.

[50]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[51]  A Tales,et al.  Mismatch negativity in the visual modality. , 1999, Neuroreport.

[52]  J. Fodor.  The Modularity of mind. An essay on faculty psychology , 1986 .

[53]  Mikko Sams,et al.  Absolute Pitch and Event-Related Brain Potentials , 1993 .

[54]  O. Reiser,et al.  Principles Of Gestalt Psychology , 1936 .

[55]  R. Näätänen,et al.  Intermodal selective attention. II. Effects of attentional load on processing of auditory and visual stimuli in central space. , 1992, Electroencephalography and clinical neurophysiology.

[56]  K. Koffka.  Principles Of Gestalt Psychology , 1936 .

[57]  Béatrice de Gelder,et al.  Exploring the relation between mcgurk interference and ventriloquism , 1994, ICSLP.

[58]  W R Thurlow,et al.  Certain Determinants of the “Ventriloquism Effect” , 1973, Perceptual and motor skills.

[59]  Cécile Colin,et al.  Rules of intersensory integration in spatial scene analysis and speechreading , 2001 .

[60]  K. Reinikainen,et al.  Mismatch negativity to change in spatial location of an auditory stimulus. , 1989, Electroencephalography and clinical neurophysiology.

[61]  R. Näätänen.  Attention and brain function , 1992 .

[62]  W R Thurlow,et al.  Effects of degree of visual association and angle of displacement on the "ventriloquism" effect. , 1973, Perceptual and motor skills.

[63]  M. Tervaniemi,et al.  Development of a memory trace for a complex sound in the human brain. , 1993, Neuroreport.

[64]  E. Schröger,et al.  Mismatch response of the human brain to changes in sound location , 1996, Neuroreport.

[65]  M. Radeau.  Auditory-visual spatial interaction and modularity , 1994, Current psychology of cognition = Cahiers de psychologie cognitive : CPC.

[66]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of deliberate visual attention , 2000, Perception & psychophysics.

[67]  P. Bertelson,et al.  The ventriloquist effect does not depend on the direction of automatic visual attention , 2001, Perception & psychophysics.

[68]  Albert S. Bregman,et al.  The Auditory Scene. (Book Reviews: Auditory Scene Analysis. The Perceptual Organization of Sound.) , 1990 .

[69]  M T Wallace,et al.  The visually responsive neuron and beyond: multisensory integration in cat and monkey. , 1993, Progress in brain research.

[70]  M A Meredith,et al.  Multisensory Integration , 1990 .

[71]  K. Alho,et al.  Intermodal selective attention. I. Effects on event-related potentials to lateralized auditory and visual stimuli. , 1992, Electroencephalography and clinical neurophysiology.

[72]  J. Driver.  Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading , 1996, Nature.

[73]  I. Winkler,et al.  Neural representation for the temporal structure of sound patterns. , 1995, Neuroreport.

[74]  R. Näätänen.  The Mismatch Negativity: A Powerful Tool for Cognitive Neuroscience , 1995, Ear and hearing.

[75]  H G Vaughan,et al.  Differentiation of negative event-related potentials in an auditory discrimination task. , 1990, Electroencephalography and clinical neurophysiology.

[76]  S. Jones,et al.  Memory-dependent auditory evoked potentials to change in the binaural interaction of noise signals. , 1991, Electroencephalography and clinical neurophysiology.