On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies

A growing number of brain imaging studies are being undertaken in order to better understand the contributions of multisensory processes to human behavior and perception. Many of these studies are designed on the basis of the physiological findings from single neurons in animal models, which have shown that multisensory neurons have the capacity for integrating their different sensory inputs and give rise to a product that differs significantly from either of the unisensory responses. At certain points these multisensory interactions can be superadditive, resulting in a neural response that exceeds the sum of the unisensory responses. Because of the difficulties inherent in interpreting the results of imaging large neuronal populations, superadditivity has been put forth as a stringent criterion for identifying potential sites of multisensory integration. In the present manuscript we discuss issues related to using the superadditive model in human brain imaging studies, focusing on population responses to multisensory stimuli and the relationship between single neuron measures and functional brain imaging measures. We suggest that the results of brain imaging studies be interpreted with caution in regards to multisensory integration. Future directions for imaging multisensory integration are discussed in light of the ideas presented.

[1]  Tom C T Yin,et al.  Bimodal Interactions in the Superior Colliculus of the Behaving Cat , 2002, The Journal of Neuroscience.

[2]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[3]  D. G. Albrecht,et al.  Spikes versus BOLD: what does neuroimaging tell us about neuronal activity? , 2000, Nature Neuroscience.

[4]  M. T. Wallace,et al.  Visual, auditory and somatosensory convergence in output neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal projection , 2005, Experimental Brain Research.

[5]  N. Logothetis The Underpinnings of the BOLD Functional Magnetic Resonance Imaging Signal , 2003, The Journal of Neuroscience.

[6]  M. Wallace,et al.  Multisensory integration in the superior colliculus of the alert cat. , 1998, Journal of neurophysiology.

[7]  P E Roland,et al.  Cross-Modal Transfer of Information between the Tactile and the Visual Representations in the Human Brain: A Positron Emission Tomographic Study , 1998, The Journal of Neuroscience.

[8]  G. Kovács,et al.  Visual, somatosensory and auditory modality properties along the feline suprageniculate-anterior ectosylvian sulcus/insular pathway. , 1996, Progress in brain research.

[9]  Keith J. Holyoak,et al.  Structure and Functions of the Human Prefrontal Cortex , 1996 .

[10]  M. Binder Integration of synaptic and intrinsic dendritic currents in cat spinal motoneurons , 2002, Brain Research Reviews.

[11]  Mark T. Wallace,et al.  Crossmodal spatial interactions in subcortical and cortical circuits , 2004 .

[12]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[13]  B. Stein,et al.  Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors , 1987, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[14]  M. Hallett,et al.  Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection , 2001, The Journal of Neuroscience.

[15]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[16]  E. DeYoe,et al.  A comparison of visual and auditory motion processing in human cerebral cortex. , 2000, Cerebral cortex.

[17]  M. Sams,et al.  Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. , 2003, Brain research. Cognitive brain research.

[18]  M T Wallace,et al.  Sensory organization of the superior colliculus in cat and monkey. , 1996, Progress in brain research.

[19]  W. Jiang,et al.  Two cortical areas mediate multisensory integration in superior colliculus neurons. , 2001, Journal of neurophysiology.

[20]  S. Iversen,et al.  Detection of Audio-Visual Integration Sites in Humans by Application of Electrophysiological Criteria to the BOLD Effect , 2001, NeuroImage.

[21]  R. Eckhorn,et al.  Perception-related modulations of local field potential power and coherence in primary visual cortex of awake monkey during binocular rivalry. , 2004, Cerebral cortex.

[22]  Karl J. Friston,et al.  A direct quantitative relationship between the functional properties of human and macaque V5 , 2000, Nature Neuroscience.

[23]  Barry E Stein,et al.  Neuron-specific response characteristics predict the magnitude of multisensory integration. , 2003, Journal of neurophysiology.

[24]  B. Stein,et al.  Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. , 1986, Journal of neurophysiology.

[25]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[26]  M. Wallace,et al.  Integration of multiple sensory modalities in cat cortex , 2004, Experimental Brain Research.

[27]  N. Logothetis,et al.  MR imaging in the non-human primate: studies of function and of dynamic connectivity , 2003, Current Opinion in Neurobiology.

[28]  S. Shimojo,et al.  Sound alters visual evoked potentials in humans , 2001, Neuroreport.

[29]  M. Giard,et al.  Auditory-Visual Integration during Multimodal Object Recognition in Humans: A Behavioral and Electrophysiological Study , 1999, Journal of Cognitive Neuroscience.

[30]  P. Goldman-Rakic Architecture of the Prefrontal Cortex and the Central Executive , 1995, Annals of the New York Academy of Sciences.

[31]  Y. Yen,et al.  Deactivation of Sensory-Specific Cortex by Cross-Modal Stimuli , 2002, Journal of Cognitive Neuroscience.

[32]  John J. Foxe,et al.  Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. , 2004, Cerebral cortex.

[33]  M. Wallace,et al.  Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. , 1993, Journal of neurophysiology.

[34]  Kyung K Peck,et al.  Comparison of hemodynamic response nonlinearity across primary cortical areas , 2004, NeuroImage.

[35]  Juliana Yordanova,et al.  Spatial coincidence modulates interaction between visual and somatosensory evoked potentials , 2002, Neuroreport.

[36]  B. Stein,et al.  Spatial determinants of multisensory integration in cat superior colliculus neurons. , 1996, Journal of neurophysiology.

[37]  E. Bullmore,et al.  Activation of auditory cortex during silent lipreading. , 1997, Science.

[38]  E Macaluso,et al.  Spatial and temporal factors during processing of audiovisual speech: a PET study , 2004, NeuroImage.

[39]  Jeffery A. Jones,et al.  Neural processes underlying perceptual enhancement by visual speech gestures , 2003, Neuroreport.

[40]  C. Frith,et al.  Modulation of human visual cortex by crossmodal spatial attention. , 2000, Science.

[41]  David A Lewis,et al.  Synaptic efficacy during repetitive activation of excitatory inputs in primate dorsolateral prefrontal cortex. , 2004, Cerebral cortex.

[42]  J. Aggleton,et al.  The functional anatomy of visual-tactile integration in man: a study using positron emission tomography , 2000, Neuropsychologia.

[43]  John J. Foxe,et al.  Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. , 2000, Brain research. Cognitive brain research.

[44]  Jeffery A. Jones,et al.  Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information , 2004, Journal of Cognitive Neuroscience.

[45]  A. J. King,et al.  Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus , 2004, Experimental Brain Research.

[46]  B. Stein,et al.  Interactions among converging sensory inputs in the superior colliculus. , 1983, Science.

[47]  G. Calvert Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[48]  Gregory McCarthy,et al.  Polysensory interactions along lateral temporal regions evoked by audiovisual speech. , 2003, Cerebral cortex.

[49]  M. Wallace,et al.  Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. , 2005, Journal of neurophysiology.

[50]  Mikko Sams,et al.  Integration of heard and seen speech: a factor in learning disabilities in children , 2003, Neuroscience Letters.

[51]  J. Driver,et al.  Multimodal mechanisms of attention related to rates of spatial shifting in vision and touch , 2001, Experimental Brain Research.

[52]  S. Ogawa Brain magnetic resonance imaging with contrast-dependent oxygenation , 1990 .

[53]  C. Mathiesen,et al.  Modification of activity‐dependent increases of cerebral blood flow by excitatory synaptic activity and spikes in rat cerebellar cortex , 1998, The Journal of physiology.

[54]  N. Logothetis,et al.  Neurophysiological investigation of the basis of the fMRI signal , 2001, Nature.

[55]  B. Stein,et al.  Spatial factors determine the activity of multisensory neurons in cat superior colliculus , 1986, Brain Research.

[56]  M. Wallace,et al.  Representation and integration of multiple sensory inputs in primate superior colliculus. , 1996, Journal of neurophysiology.

[57]  Katie L. McMahon,et al.  Orthographic/Phonological Facilitation of Naming Responses in the Picture–Word Task: An Event-Related fMRI Study Using Overt Vocal Responding , 2002, NeuroImage.

[58]  Riitta Hari,et al.  Audiovisual Integration of Letters in the Human Brain , 2000, Neuron.

[59]  M. Wallace,et al.  A revised view of sensory cortical parcellation , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[60]  J Driver,et al.  Selective spatial attention in vision and touch: unimodal and multimodal mechanisms revealed by PET. , 2000, Journal of neurophysiology.

[61]  P. Goldman-Rakic,et al.  Elevated neuronal density in prefrontal area 46 in brains from schizophrenic patients: Application of a three‐dimensional, stereologic counting method , 1998, The Journal of comparative neurology.

[62]  Paul J. Laurienti,et al.  Dietary Caffeine Consumption Modulates fMRI Measures , 2002, NeuroImage.

[63]  D. Tank,et al.  Brain magnetic resonance imaging with contrast dependent on blood oxygenation. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[64]  Tarek A. Yousry,et al.  fMRI signal increases and decreases in cortical areas during small-field optokinetic stimulation and central fixation , 2002, Experimental Brain Research.

[65]  Riitta Hari,et al.  Activation of the human posterior parietal and temporoparietal cortices during audiotactile interaction , 2003, NeuroImage.

[66]  Joseph A Maldjian,et al.  Cross‐modal sensory processing in the anterior cingulate and medial prefrontal cortices , 2003, Human brain mapping.

[67]  M. Margulis,et al.  Temporal integration can readily switch between sublinear and supralinear summation. , 1998, Journal of neurophysiology.

[68]  Martin Lauritzen,et al.  Brain Function and Neurophysiological Correlates of Signals Used in Functional Neuroimaging , 2003, The Journal of Neuroscience.