Organization, Maturation, and Plasticity of Multisensory Integration: Insights from Computational Modeling Studies

In this paper, we present two neural network models – devoted to two specific and widely investigated aspects of multisensory integration – in order to evidence the potentialities of computational models to gain insight into the neural mechanisms underlying organization, development, and plasticity of multisensory integration in the brain. The first model considers visual–auditory interaction in a midbrain structure named superior colliculus (SC). The model is able to reproduce and explain the main physiological features of multisensory integration in SC neurons and to describe how SC integrative capability – not present at birth – develops gradually during postnatal life depending on sensory experience with cross-modal stimuli. The second model tackles the problem of how tactile stimuli on a body part and visual (or auditory) stimuli close to the same body part are integrated in multimodal parietal neurons to form the perception of peripersonal (i.e., near) space. The model investigates how the extension of peripersonal space – where multimodal integration occurs – may be modified by experience such as use of a tool to interact with the far space. The utility of the modeling approach relies on several aspects: (i) The two models, although devoted to different problems and simulating different brain regions, share some common mechanisms (lateral inhibition and excitation, non-linear neuron characteristics, recurrent connections, competition, Hebbian rules of potentiation and depression) that may govern more generally the fusion of senses in the brain, and the learning and plasticity of multisensory integration. (ii) The models may help interpretation of behavioral and psychophysical responses in terms of neural activity and synaptic connections. (iii) The models can make testable predictions that can help guiding future experiments in order to validate, reject, or modify the main assumptions.

[1]  B. Stein,et al.  Sources of subcortical projections to the superior colliculus in the cat , 1979, The Journal of comparative neurology.

[2]  G. Rizzolatti,et al.  Afferent properties of periarcuate neurons in macaque monkeys. I. Somatosensory responses , 1981, Behavioural Brain Research.

[3]  G. Rizzolatti,et al.  Afferent properties of periarcuate neurons in macaque monkeys. II. Visual responses , 1981, Behavioural Brain Research.

[4]  J. K. Harting,et al.  The Mammalian Superior Colliculus: Studies of Its Morphology and Connections , 1984 .

[5]  B. Stein,et al.  Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. , 1986, Journal of neurophysiology.

[6]  C. Caltagirone,et al.  Contralateral and ipsilateral tactile extinction in patients with right and left focal brain damage. , 1989, The International journal of neuroscience.

[7]  G. Gainotti,et al.  Contralateral and ipsilateral disorders of visual attention in patients with unilateral brain damage. , 1990, Journal of neurology, neurosurgery, and psychiatry.

[8]  M. Wallace,et al.  Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. , 1993, Journal of neurophysiology.

[9]  M T Wallace,et al.  Cross-modal synthesis in the midbrain depends on input from cortex. , 1994, Journal of neurophysiology.

[10]  Terrence J. Sejnowski,et al.  Spatial Representations in the Parietal Cortex May Use Basis Functions , 1994, NIPS.

[11]  G. Rizzolatti,et al.  Coding of peripersonal space in inferior premotor cortex (area F4). , 1996, Journal of neurophysiology.

[12]  B. Stein,et al.  Spatial determinants of multisensory integration in cat superior colliculus neurons. , 1996, Journal of neurophysiology.

[13]  M. Tanaka,et al.  Coding of modified body schema during tool use by macaque postcentral neurones. , 1996, Neuroreport.

[14]  C. Gross,et al.  Visuospatial properties of ventral premotor cortex. , 1997, Journal of neurophysiology.

[15]  G Rizzolatti,et al.  The Space Around Us , 1997, Science.

[16]  M T Wallace,et al.  Development of Multisensory Neurons and Multisensory Integration in Cat Superior Colliculus , 1997, The Journal of Neuroscience.

[17]  M T Wallace,et al.  Mechanisms of within- and cross-modality suppression in the superior colliculus. , 1997, Journal of neurophysiology.

[18]  Elisabetta Làdavas,et al.  Seeing where your hands are , 1997, Nature.

[19]  T. Sejnowski,et al.  Spatial Transformations in the Parietal Cortex Using Basis Functions , 1997, Journal of Cognitive Neuroscience.

[20]  E. Rolls,et al.  Neural networks and brain function , 1998 .

[21]  Giuseppe di Pellegrino,et al.  Neuropsychological Evidence of an Integrated Visuotactile Representation of Peripersonal Space in Humans , 1998, Journal of Cognitive Neuroscience.

[22]  C. Spence,et al.  Cross-modal links in spatial attention. , 1998, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[23]  M. Goldberg,et al.  Ventral intraparietal area of the macaque: congruent visual and somatic response properties. , 1998, Journal of neurophysiology.

[24]  M. Wallace,et al.  Multisensory integration in the superior colliculus of the alert cat. , 1998, Journal of neurophysiology.

[25]  C. Gross,et al.  A neuronal representation of the location of nearby sounds , 1999, Nature.

[26]  C. Frith,et al.  Modulation of human visual cortex by crossmodal spatial attention. , 2000, Science.

[27]  Thomas J. Anastasio,et al.  Using Bayes' Rule to Model Multisensory Enhancement in the Superior Colliculus , 2000, Neural Computation.

[28]  C. Spence,et al.  Multisensory perception: Beyond modularity and convergence , 2000, Current Biology.

[29]  A. Iriki,et al.  Acquisition and development of monkey tool-use: behavioral and kinematic analyses. , 2000, Canadian journal of physiology and pharmacology.

[30]  A Farnè,et al.  Dynamic size‐change of hand peripersonal space following tool use , 2000, Neuroreport.

[31]  M T Wallace,et al.  Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. , 2000, Journal of neurophysiology.

[32]  T. Sejnowski,et al.  Simulating a lesion in a basis function model of spatial representations: comparison with hemineglect. , 2001, Psychological review.

[33]  W. Jiang,et al.  Two cortical areas mediate multisensory integration in superior colliculus neurons. , 2001, Journal of neurophysiology.

[34]  K. Zilles,et al.  Polymodal Motion Processing in Posterior Parietal and Premotor Cortex A Human fMRI Study Strongly Implies Equivalencies between Humans and Monkeys , 2001, Neuron.

[35]  G. Calvert Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[36]  Martin Eimer,et al.  Crossmodal links in spatial attention are mediated by supramodal control processes: evidence from event-related potentials. , 2002, Psychophysiology.

[37]  N. Bolognini,et al.  Enhancement of visual perception by crossmodal visuo-auditory interaction , 2002, Experimental Brain Research.

[38]  Jon Driver,et al.  The electrophysiology of tactile extinction: ERP correlates of unconscious somatosensory processing , 2002, Neuropsychologia.

[39]  Kamel Belkacem-Boussaid,et al.  Multimodality in the superior colliculus: an information theoretic analysis. , 2002, Brain research. Cognitive brain research.

[40]  E. Ldavas,et al.  Auditory Peripersonal Space in Humans , 2002, Journal of Cognitive Neuroscience.

[41]  Mark T. Wallace,et al.  The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus , 2002, Experimental Brain Research.

[42]  Alexandre Pouget,et al.  A computational perspective on the neural basis of multisensory spatial representations , 2002, Nature Reviews Neuroscience.

[43]  Barry E Stein,et al.  Neuron-specific response characteristics predict the magnitude of multisensory integration. , 2003, Journal of neurophysiology.

[44]  Thomas J. Anastasio,et al.  Modeling Cross-Modal Enhancement and Modality-Specific Suppression in Multisensory Neurons , 2003, Neural Computation.

[45]  Paul E Patton,et al.  A Two-Stage Unsupervised Learning Algorithm Reproduces Multisensory Enhancement in a Neural Network Model of the Corticotectal System , 2003, The Journal of Neuroscience.

[46]  John C. Middlebrooks,et al.  Directional sensitivity of neurons in the primary auditory (AI) cortex: effects of sound-source intensity level. , 2003, Journal of neurophysiology.

[47]  D. Amit,et al.  The eye and the hand: neural mechanisms and network models for oculomanual coordination in parietal cortex. , 2003, Cerebral cortex.

[48]  A. Diederich,et al.  Why aren’t all deep superior colliculus neurons multisensory? A Bayes’ ratio analysis , 2004, Cognitive, Affective, & Behavioral Neuroscience.

[49]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[50]  Barry E Stein,et al.  Visual Experience Is Necessary for the Development of Multisensory Integration , 2004, The Journal of Neuroscience.

[51]  C. Spence,et al.  Extending or projecting peripersonal space with tools? Multisensory interactions highlight only the distal and proximal ends of tools , 2004, Neuroscience Letters.

[52]  Nadia Bolognini,et al.  Audiovisual Integration in Patients with Visual Deficit , 2005, Journal of Cognitive Neuroscience.

[53]  T. Stanford,et al.  Evaluating the Operations Underlying Multisensory Integration in the Cat Superior Colliculus , 2005, The Journal of Neuroscience.

[54]  John J. Foxe,et al.  Multisensory contributions to low-level, ‘unisensory’ processing , 2005, Current Opinion in Neurobiology.

[55]  E. Macaluso,et al.  Multisensory spatial interactions: a window onto functional integration in the human brain , 2005, Trends in Neurosciences.

[56]  Atsushi Iriki,et al.  Shaping multisensory action–space with tools: evidence from patients with cross-modal extinction , 2005, Neuropsychologia.

[57]  A. Pouget,et al.  Reference frames for representing visual and tactile locations in parietal cortex , 2005, Nature Neuroscience.

[58]  M. Wallace,et al.  Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. , 2005, Journal of neurophysiology.

[59]  Wan Jiang,et al.  Neonatal cortical ablation disrupts multisensory development in superior colliculus. , 2006, Journal of neurophysiology.

[60]  Jon Driver,et al.  Neural correlates of crossmodal visual-tactile extinction and of tactile awareness revealed by fMRI in a right-hemisphere stroke patient , 2006, Neuropsychologia.

[61]  S. Ichinose,et al.  Extension of Corticocortical Afferents into the Anterior Bank of the Intraparietal Sulcus by Tool-use Training in Adult Monkeys , 2005 .

[62]  Argye E. Hillis,et al.  Neural Correlates of Modality-specific Spatial Extinction , 2006, Journal of Cognitive Neuroscience.

[63]  Nicholas P. Holmes,et al.  Tool use changes multisensory interactions in seconds: evidence from the crossmodal congruency task , 2007, Experimental Brain Research.

[64]  M. Costantini,et al.  Temporal dynamics of visuo-tactile extinction within and between hemispaces. , 2007, Neuropsychology.

[65]  Wan Jiang,et al.  Multisensory orientation behavior is disrupted by neonatal cortical ablation. , 2007, Journal of neurophysiology.

[66]  Nicholas P. Holmes,et al.  Tool-Use: Capturing Multisensory Spatial Attention or Extending Multisensory Peripersonal Space? , 2007, Cortex.

[67]  M. Bassolino,et al.  PSYCHOLOGICAL SCIENCE Research Article Extended Multisensory Space in Blind Cane Users , 2022 .

[68]  Andrea Serino,et al.  Dynamic Size-Change of Peri-Hand Space Following Tool-Use: Determinants and Spatial Characteristics Revealed Through Cross-Modal Extinction , 2007, Cortex.

[69]  Ehud Zohary,et al.  Is That Near My Hand? Multisensory Representation of Peripersonal Space in Human Intraparietal Sulcus , 2007, The Journal of Neuroscience.

[70]  Terrence R Stanford,et al.  A Model of the Neural Mechanisms Underlying Multisensory Integration in the Superior Colliculus , 2007, Perception.

[71]  Terrence R Stanford,et al.  Cortex Mediates Multisensory But Not Unisensory Integration in Superior Colliculus , 2007, The Journal of Neuroscience.

[72]  P. Haggard,et al.  Viewing the body modulates tactile receptive fields , 2007, Experimental Brain Research.

[73]  Sidney S. Simon,et al.  Merging of the Senses , 2008, Front. Neurosci..

[74]  Mauro Ursino,et al.  Multisensory integration in the superior colliculus: a neural network model , 2009, Journal of Computational Neuroscience.

[75]  Mauro Ursino,et al.  A theoretical study of multisensory integration in the superior colliculus by a neural network model , 2008, Neural Networks.

[76]  Khurshid Ahmad,et al.  Modeling Multisensory Enhancement with Self-organizing Maps , 2009, Front. Comput. Neurosci..

[77]  T. Stanford,et al.  Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness , 2009, Experimental Brain Research.

[78]  Benjamin A. Rowland,et al.  Multisensory Integration in the Superior Colliculus Requires Synergy among Corticocollicular Inputs , 2009, The Journal of Neuroscience.

[79]  Mauro Ursino,et al.  Neural bases of peri-hand space plasticity through tool-use: Insights from a combined computational–experimental approach , 2010, Neuropsychologia.

[80]  Mauro Ursino,et al.  Visuotactile Representation of Peripersonal Space: A Neural Network Study , 2010, Neural Computation.

[81]  Andrea Serino,et al.  Everyday use of the computer mouse extends peripersonal space representation , 2010, Neuropsychologia.

[82]  Benjamin A. Rowland,et al.  An Emergent Model of Multisensory Integration in Superior Colliculus Neurons , 2010, Front. Integr. Neurosci..