Cortical network underlying audiovisual semantic integration and modulation of attention: An fMRI and graph-based study

Many neuroimaging and electrophysiology studies have suggested that semantic integration as a high-level cognitive process involves various cortical regions and is modulated by attention. However, the cortical network specific to semantic integration and the modulatory mechanism of attention remain unclear. Here, we designed an fMRI experiment using “bimodal stimulus” to extract information regarding the cortical activation related to the effects of semantic integration with and without attention, and then analyzed the characteristics of the cortical network and the modulating effect of attention on semantic integration. To further investigate the related cortical regions, we constructed a functional brain network for processing attended AV stimuli to evaluate the nodal properties using a graph-based method. The results of the fMRI and graph-based analyses showed that the semantic integration with attention activated the anterior temporal lobe (ATL), temporoparietal junction (TPJ), and frontoparietal cortex, with the ATL showing the highest nodal degree and efficiency; in contrast, semantic integration without attention involved a relatively small cortical network, including the posterior superior temporal gyrus (STG), Heschl’s gyrus (HG), and precentral gyrus. These results indicated that semantic integration is a complex cognitive process that occurs not only in the attended condition but also in the unattended condition, and that attention could modulate the distribution of cortical networks related to semantic integration. We suggest that semantic integration with attention is a conscious process and needs a wide cortical network working together, in which the ATL plays the role of a central hub; in contrast, semantic integration without attention is a pre-attentive process and involves a relatively smaller cortical network, in which the HG may play an important role. Our study will provide valuable insights into semantic integration and will be useful for investigations on multisensory integration and attention mechanism at multiple processing stages and levels within the cortical hierarchy.

[1]  E. Jefferies,et al.  Amodal semantic representations depend on both anterior temporal lobes: Evidence from repetitive transcranial magnetic stimulation , 2010, Neuropsychologia.

[2]  T. Rogers,et al.  The neural and computational bases of semantic cognition , 2016, Nature Reviews Neuroscience.

[3]  Jan Theeuwes,et al.  Pip and pop: nonspatial auditory signals improve spatial visual search. , 2008, Journal of experimental psychology. Human perception and performance.

[4]  Albert Costa,et al.  Brain networks involved in accented speech processing , 2019, Brain and Language.

[5]  U. Noppeney,et al.  Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. , 2010, Cerebral cortex.

[6]  D. Senkowski,et al.  The multifaceted interplay between attention and multisensory integration , 2010, Trends in Cognitive Sciences.

[7]  J. Driver,et al.  Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and Judgments , 2008, Neuron.

[8]  Joseph A Maldjian,et al.  Cross‐modal sensory processing in the anterior cingulate and medial prefrontal cortices , 2003, Human brain mapping.

[9]  E. Rolls,et al.  A Neurodynamical cortical model of visual attention and invariant object recognition , 2004, Vision Research.

[10]  B. Stein,et al.  Spatial determinants of multisensory integration in cat superior colliculus neurons. , 1996, Journal of neurophysiology.

[11]  Grant M. Walker,et al.  Neuroanatomical dissociation for taxonomic and thematic knowledge in the human brain , 2011, Proceedings of the National Academy of Sciences.

[12]  Matthew A. Lambon Ralph,et al.  Unveiling the dynamic interplay between the hub- and spoke-components of the brain's semantic system and its impact on human behaviour , 2019, NeuroImage.

[13]  J. Duncan.  The Structure of Cognition: Attentional Episodes in Mind and Brain , 2013, Neuron.

[14]  E. Miller,et al.  An integrative theory of prefrontal cortex function. , 2001, Annual review of neuroscience.

[15]  Andreas Horn,et al.  The structural–functional connectome and the default mode network of the human brain , 2014, NeuroImage.

[16]  M. Woldorff,et al.  Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? , 2006, Cerebral cortex.

[17]  Paul J. Laurienti,et al.  Frontiers in Aging Neuroscience Aging Neuroscience , 2022 .

[18]  J. Driver,et al.  Sound-Induced Enhancement of Low-Intensity Vision: Multisensory Influences on Human Sensory-Specific Cortices and Thalamic Bodies Relate to Perceptual Enhancement of Visual Detection Sensitivity , 2010, The Journal of Neuroscience.

[19]  T. Hackett,et al.  Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[20]  Lars Muckli,et al.  Cortical Plasticity of Audio–Visual Object Representations , 2008, Cerebral cortex.

[21]  Matthew A. Lambon Ralph,et al.  Neurocognitive insights on conceptual knowledge and its breakdown , 2014 .

[22]  M. L. Lambon Ralph,et al.  Category-Specific versus Category-General Semantic Impairment Induced by Transcranial Magnetic Stimulation , 2010, Current Biology.

[23]  Yuanqing Li,et al.  Reproducibility and Discriminability of Brain Patterns of Semantic Categories Enhanced by Congruent Audiovisual Stimuli , 2011, PloS one.

[24]  M. Corbetta,et al.  Control of goal-directed and stimulus-driven attention in the brain , 2002, Nature Reviews Neuroscience.

[25]  Rutvik H. Desai,et al.  The neurobiology of semantic memory , 2011, Trends in Cognitive Sciences.

[26]  Jia Huang,et al.  Neural Correlates of Audiovisual Sensory Integration , 2018, Neuropsychology.

[27]  Sophie M. Wuerger,et al.  Interactions between Auditory and Visual Semantic Stimulus Classes: Evidence for Common Processing Networks for Speech and Body Actions , 2011, Journal of Cognitive Neuroscience.

[28]  J. Rauschecker,et al.  Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing , 2009, Nature Neuroscience.

[29]  U. Noppeney,et al.  Distinct Functional Contributions of Primary Sensory and Association Areas to Audiovisual Integration in Object Categorization , 2010, The Journal of Neuroscience.

[30]  Christof Karmonik,et al.  Characterization of functional brain connectivity towards optimization of music selection for therapy: a fMRI study , 2019, The International journal of neuroscience.

[31]  Kayoko Okada,et al.  Two cortical mechanisms support the integration of visual and auditory speech: A hypothesis and preliminary data , 2009, Neuroscience Letters.

[32]  Yuanqing Li,et al.  Selective Audiovisual Semantic Integration Enabled by Feature-Selective Attention , 2016, Scientific Reports.

[33]  Emily J. Mayberry,et al.  Coherent concepts are computed in the anterior temporal lobes , 2010, Proceedings of the National Academy of Sciences.

[34]  Joost X. Maier,et al.  Natural, Metaphoric, and Linguistic Auditory Direction Signals Have Distinct Influences on Visual Motion Processing , 2009, The Journal of Neuroscience.

[35]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[36]  Y. Yen,et al.  Deactivation of Sensory-Specific Cortex by Cross-Modal Stimuli , 2002, Journal of Cognitive Neuroscience.

[37]  C. Schroeder,et al.  Neuronal Oscillations and Multisensory Interaction in Primary Auditory Cortex , 2007, Neuron.

[38]  Matthew A. Lambon Ralph,et al.  Neurocognitive insights on conceptual knowledge and its breakdown , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[39]  Ryan A. Stevenson,et al.  Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects , 2007, Experimental Brain Research.

[40]  Rebecca L. Jackson,et al.  The Nature and Neural Correlates of Semantic Association versus Conceptual Similarity , 2015, Cerebral cortex.

[41]  Marty G. Woldorff,et al.  Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity , 2005, Journal of Cognitive Neuroscience.

[42]  S. Dehaene,et al.  Unconscious task set priming with phonological and semantic tasks , 2013, Consciousness and Cognition.

[43]  Hongtao Yu,et al.  The informativity of sound modulates crossmodal facilitation of visual discrimination: a fMRI study , 2017, Neuroreport.

[44]  Bradford Z. Mahon,et al.  A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content , 2008, Journal of Physiology-Paris.

[45]  T. Münte,et al.  Audiovisual speech integration in the superior temporal region is dysfunctional in dyslexia , 2017, Neuroscience.

[46]  Susan L. Whitfield-Gabrieli,et al.  Conn: A Functional Connectivity Toolbox for Correlated and Anticorrelated Brain Networks , 2012, Brain Connect..

[47]  Teija Kujala,et al.  Word Semantics Is Processed Even without Attentional Effort , 2009, Journal of Cognitive Neuroscience.

[48]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[49]  Ryan A. Stevenson,et al.  Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition , 2009, NeuroImage.

[50]  T. Rogers,et al.  Colour knowledge in semantic dementia: It is not all black and white , 2007, Neuropsychologia.

[51]  N. Logothetis,et al.  Visual modulation of neurons in auditory cortex. , 2008, Cerebral cortex.

[52]  R. Campbell,et al.  Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex , 2000, Current Biology.

[53]  Wolfgang Ellermeier,et al.  Neural correlates of audio‐visual object recognition: Effects of implicit spatial congruency , 2012, Human brain mapping.

[54]  G. Davis,et al.  Unconscious priming of task sets: The role of spatial attention , 2012, Attention, perception & psychophysics.

[55]  O. Sporns,et al.  Network centrality in the human functional connectome. , 2012, Cerebral cortex.

[56]  J. Theeuwes,et al.  Poke and pop: Tactile–visual synchrony increases visual saliency , 2009, Neuroscience Letters.

[57]  Lars Nyberg,et al.  Cortical regions underlying successful encoding of semantically congruent and incongruent associations between common auditory and visual objects , 2011, Neuroscience Letters.

[58]  J. Qiu,et al.  Prefrontal sensitivity to changes in language form and semantic content during speech production , 2019, Brain and Language.

[59]  Daniel Mirman,et al.  Taxonomic and thematic semantic systems. , 2017, Psychological bulletin.

[60]  E. Bullmore,et al.  Response amplification in sensory-specific cortices during crossmodal binding. , 1999, Neuroreport.

[61]  Matthew A. Lambon Ralph,et al.  Convergent Connectivity and Graded Specialization in the Rostral Human Temporal Lobe as Revealed by Diffusion-Weighted Imaging Probabilistic Tractography , 2012, Journal of Cognitive Neuroscience.

[62]  Adolfo M García,et al.  How meaning unfolds in neural time: Embodied reactivations can precede multimodal semantic effects during language processing , 2019, NeuroImage.