A Neural Network Model Can Explain Ventriloquism Aftereffect and Its Generalization across Sound Frequencies

Exposure to synchronous but spatially disparate auditory and visual stimuli produces a perceptual shift of sound location towards the visual stimulus (ventriloquism effect). After adaptation to a ventriloquism situation, enduring sound shift is observed in the absence of the visual stimulus (ventriloquism aftereffect). Experimental studies report opposing results as to aftereffect generalization across sound frequencies varying from aftereffect being confined to the frequency used during adaptation to aftereffect generalizing across some octaves. Here, we present an extension of a model of visual-auditory interaction we previously developed. The new model is able to simulate the ventriloquism effect and, via Hebbian learning rules, the ventriloquism aftereffect and can be used to investigate aftereffect generalization across frequencies. The model includes auditory neurons coding both for the spatial and spectral features of the auditory stimuli and mimicking properties of biological auditory neurons. The model suggests that different extent of aftereffect generalization across frequencies can be obtained by changing the intensity of the auditory stimulus that induces different amounts of activation in the auditory layer. The model provides a coherent theoretical framework to explain the apparently contradictory results found in the literature. Model mechanisms and hypotheses are discussed in relation to neurophysiological and psychophysical data.

[1]  Fabrizio Leo,et al.  Independent mechanisms for ventriloquism and multisensory integration as revealed by theta‐burst stimulation , 2010, The European journal of neuroscience.

[2]  J. Lewald Rapid adaptation to auditory-visual spatial disparity. , 2002, Learning & memory.

[3]  T. Hackett,et al.  Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[4]  S. Hillyard,et al.  Neural Basis of the Ventriloquist Illusion , 2007, Current Biology.

[5]  Lisa A. de la Mothe,et al.  A comparison of neuron response properties in areas A1 and CM of the marmoset monkey auditory cortex: tones and broadband noise. , 2005, Journal of neurophysiology.

[6]  G. Recanzone,et al.  Effects of stimulus azimuth and intensity on the single-neuron activity in the auditory cortex of the alert macaque monkey. , 2006, Journal of neurophysiology.

[7]  Paul Bertelson,et al.  The aftereffects of ventriloquism: are they sound-frequency specific? , 2003, Acta psychologica.

[8]  G. Recanzone Interactions of auditory and visual stimuli in space and time , 2009, Hearing Research.

[9]  Elisa Magosso,et al.  Integrating Information From Vision and Touch: A Neural Network Modeling Study , 2010, IEEE Transactions on Information Technology in Biomedicine.

[10]  Nadia Bolognini,et al.  Multisensory-Mediated Auditory Localization , 2006, Perception.

[11]  G. Recanzone,et al.  Frequency and intensity response properties of single neurons in the auditory cortex of the behaving macaque monkey. , 2000, Journal of neurophysiology.

[12]  Mauro Ursino,et al.  A Neural Network Model of Ventriloquism Effect and Aftereffect , 2012, PloS one.

[13]  John J. Foxe,et al.  Multisensory contributions to low-level, ‘unisensory’ processing , 2005, Current Opinion in Neurobiology.

[14]  Paul Bertelson,et al.  The aftereffects of ventriloquism: generalization across sound-frequencies. , 2005, Acta psychologica.

[15]  David R. Wozny,et al.  Recalibration of Auditory Space following Milliseconds of Cross-Modal Discrepancy , 2011, The Journal of Neuroscience.

[16]  Benjamin A. Rowland,et al.  Hebbian mechanisms help explain development of multisensory integration in the superior colliculus: a neural network model , 2012, Biological Cybernetics.

[17]  Paul Bertelson,et al.  The aftereffects of ventriloquism: Patterns of spatial generalization , 2006, Perception & psychophysics.

[18]  Ilja Frissen,et al.  The aftereffects of ventriloquism: the time course of the visual recalibration of auditory localization. , 2012, Seeing and perceiving.

[19]  D. H. Warren,et al.  Immediate perceptual response to intersensory discrepancy. , 1980, Psychological bulletin.

[20]  G. Recanzone Rapidly induced auditory plasticity: the ventriloquism aftereffect. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[21]  G H Recanzone,et al.  Correlation between the activity of single auditory cortical neurons and sound-localization behavior in the macaque monkey. , 2000, Journal of neurophysiology.

[22]  Benjamin A. Rowland,et al.  An Emergent Model of Multisensory Integration in Superior Colliculus Neurons , 2010, Front. Integr. Neurosci..

[23]  P. Bertelson,et al.  Cross-modal bias and perceptual fusion with auditory-visual spatial discordance , 1981, Perception & psychophysics.

[24]  Christoph Kayser,et al.  Multisensory interactions in primate auditory cortex: fMRI and electrophysiology , 2009, Hearing Research.

[25]  G. Recanzone,et al.  Temporal and spatial dependency of the ventriloquism effect , 2001, Neuroreport.

[26]  Kenneth D. Miller,et al.  The Role of Constraints in Hebbian Learning , 1994, Neural Computation.

[27]  G. Aschersleben,et al.  Automatic visual bias of perceived auditory location , 1998 .

[28]  Fabrizio Leo,et al.  Cross-modal localization in hemianopia: new insights on multisensory integration. , 2008, Brain : a journal of neurology.

[29]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[30]  Mauro Ursino,et al.  Neural bases of peri-hand space plasticity through tool-use: Insights from a combined computational–experimental approach , 2010, Neuropsychologia.

[31]  Timothy M. Woods,et al.  Visually Induced Plasticity of Auditory Spatial Perception in Macaques , 2004, Current Biology.

[32]  M. Wallace,et al.  Visual Localization Ability Influences Cross-Modal Bias , 2003, Journal of Cognitive Neuroscience.

[33]  Benjamin A. Rowland,et al.  A computational study of multisensory maturation in the superior colliculus (SC) , 2011, Experimental Brain Research.

[34]  L. Aitkin,et al.  Azimuthal sensitivity of neurons in primary auditory cortex of cats. I. Types of sensitivity and the effects of variations in stimulus parameters. , 1990, Journal of neurophysiology.

[35]  G H Recanzone,et al.  Spatial processing in the auditory cortex of the macaque monkey. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[36]  Ilja Frissen,et al.  Visual recalibration of auditory spatial perception: two separate neural circuits for perceptual learning , 2009, The European journal of neuroscience.

[37]  Mauro Ursino,et al.  Multisensory integration in the superior colliculus: a neural network model , 2009, Journal of Computational Neuroscience.

[38]  M. Radeau,et al.  Signal Intensity, Task Context, and Auditory-Visual Interactions , 1985, Perception.

[39]  Mauro Ursino,et al.  A theoretical study of multisensory integration in the superior colliculus by a neural network model , 2008, Neural Networks.

[40]  B. Stein,et al.  The Merging of the Senses , 1993 .

[41]  M. Wallace,et al.  Unifying multisensory signals across time and space , 2004, Experimental Brain Research.

[42]  P. Mamassian,et al.  Multisensory processing in review: from physiology to behaviour. , 2010, Seeing and perceiving.

[43]  A. Ghazanfar,et al.  Is neocortex essentially multisensory? , 2006, Trends in Cognitive Sciences.

[44]  Mauro Ursino,et al.  Visuotactile Representation of Peripersonal Space: A Neural Network Study , 2010, Neural Computation.