The role of the left and right inferior frontal gyrus in processing metaphoric and unrelated co-speech gestures

Gestures are an integral part of in-person conversations and complement the meaning of the speech they accompany. The neural processing of co-speech gestures is supported by a mostly left-lateralized network of fronto-temporal regions. However, in contrast to iconic gestures, metaphoric as well as unrelated gestures have been found to more strongly engage the left and right inferior frontal gyrus (IFG), respectively. With this study, we conducted the first systematic comparison of all three types of gestures and resulting potential laterality effects. During collection of functional imaging data, 74 subjects were presented with 5-second videos of abstract speech with related metaphoric gestures, concrete speech with related iconic gestures and concrete speech with unrelated gestures. They were asked to judge whether the content of the speech and gesture matched or not. Differential contrasts revealed that both abstract related and concrete unrelated compared to concrete related stimuli elicited stronger activation of the bilateral IFG. Analyses of lateralization indices for IFG activation further showed a left hemispheric dominance for metaphoric gestures and a right hemispheric dominance for unrelated gestures. Our results give support to the hypothesis that the bilateral IFG is activated specifically when processing load for speech-gesture combinations is high. In addition, laterality effects indicate a stronger involvement of the right IFG in mismatch detection and conflict processing, whereas the left IFG performs the actual integration of information from speech and gesture.

[1]  C. Burgess,et al.  Neurocognitive Mechanisms Underlying Metaphor Comprehension and Other Figurative Language , 1996 .

[2]  Isabell Wartenburger,et al.  Show your hands — Are you really clever? Reasoning, gesture production, and intelligence , 2011 .

[3]  Maria Montefinese.  Semantic representation of abstract and concrete words: a minireview of neural evidence. , 2019, Journal of neurophysiology.

[4]  Justine Cassell,et al.  Communicative Effects of Speech-Mismatched Gestures , 1994 .

[5]  K. Willmes,et al.  Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures , 2009, Neuropsychologia.

[6]  T. Kircher,et al.  Semantic relation vs. surprise: The differential effects of related and unrelated co-verbal gestures on neural encoding and subsequent recognition , 2014, Brain Research.

[7]  Joseph A Maldjian,et al.  Precentral gyrus discrepancy in electronic versions of the Talairach atlas , 2004, NeuroImage.

[8]  Michael Erb,et al.  Laterality in metaphor processing: Lack of evidence from functional magnetic resonance imaging for the right hemisphere theory , 2007, Brain and Language.

[9]  T. Kircher,et al.  A brief self-rating scale for the assessment of individual differences in gesture perception and production , 2015 .

[10]  Michael S. Gaffrey,et al.  Activity and functional connectivity of inferior frontal cortex associated with response conflict. , 2005, Brain research. Cognitive brain research.

[11]  Dustin Scheinost,et al.  Cluster failure or power failure? Evaluating sensitivity in cluster-level inference , 2019, NeuroImage.

[12]  Richard P. Cooper,et al.  Is there a semantic system for abstract words? , 2013, Front. Hum. Neurosci..

[13]  B. Straube,et al.  Dynamic Causal Modelling suggests impaired effective connectivity in patients with schizophrenia spectrum disorders during gesture-speech integration , 2019, Schizophrenia Research.

[14]  Jeremy I. Skipper,et al.  Co‐speech gestures influence neural activity in brain regions associated with processing semantic information , 2009, Human brain mapping.

[15]  Skipper Seabold,et al.  Statsmodels: Econometric and Statistical Modeling with Python , 2010, SciPy.

[16]  D. Margulies,et al.  Functional Segregation of the Right Inferior Frontal Gyrus: Evidence From Coactivation-Based Parcellation , 2019, Cerebral cortex.

[17]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[18]  K. Hugdahl,et al.  Lateralization of cognitive processes in the brain. , 2000, Acta psychologica.

[19]  Martha W. Alibali,et al.  Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production , 2007 .

[20]  Jonathan D. Power,et al.  Recent progress and outstanding issues in motion correction in resting state fMRI , 2015, NeuroImage.

[21]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[22]  E. Rolls.  The orbitofrontal cortex and emotion in health and disease, including depression , 2017, Neuropsychologia.

[23]  Michael Erb,et al.  Where in the brain is nonliteral language? A coordinate-based meta-analysis of functional magnetic resonance imaging studies , 2012, NeuroImage.

[24]  T. Kircher,et al.  Evidence for gesture-speech mismatch detection impairments in schizophrenia , 2019, Psychiatry Research.

[25]  Andreas Hennenlotter,et al.  Neural correlates of the processing of co-speech gestures , 2008, NeuroImage.

[26]  Asli Ozyurek.  Hearing and seeing meaning in speech and gesture: insights from brain and behaviour , 2014 .

[27]  Klaas E. Stephan,et al.  Mechanisms of hemispheric lateralization: Asymmetric interhemispheric recruitment in the face perception network , 2016, NeuroImage.

[28]  Wolfgang Grodd,et al.  An fMRI task battery for assessing hemispheric language dominance in children , 2006, NeuroImage.

[29]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[30]  Paul J. Laurienti,et al.  An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets , 2003, NeuroImage.

[31]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.

[32]  C. Kennard,et al.  Functional role of the supplementary and pre-supplementary motor areas , 2008, Nature Reviews Neuroscience.

[33]  Yang Yang,et al.  Neural Systems Underlying Emotional and Non-emotional Interference Processing: An ALE Meta-Analysis of Functional Neuroimaging Studies , 2016, Front. Behav. Neurosci..

[34]  A. Saygin,et al.  Distinct representations in occipito-temporal, parietal, and premotor cortex during action perception revealed by fMRI and computational modeling , 2019, Neuropsychologia.

[35]  Peter,et al.  Semantic Unification , 2008 .

[36]  Tilo Kircher,et al.  A Supramodal Neural Network for Speech and Gesture Semantics: An fMRI Study , 2012, PloS one.

[37]  G. Rizzolatti,et al.  Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study , 2001, The European journal of neuroscience.

[38]  Mark W. Woolrich,et al.  FSL , 2012, NeuroImage.

[39]  B. Straube,et al.  Transcranial Direct Current Stimulation Improves Semantic Speech–Gesture Matching in Patients With Schizophrenia Spectrum Disorder , 2018, Schizophrenia bulletin.

[40]  Arne Nagels,et al.  Feeling addressed! The role of body orientation and co‐speech gesture in social communication , 2015, Human brain mapping.

[41]  T. Kircher,et al.  Feeling addressed! The neural processing of social communicative cues in patients with major depression , 2020, Human brain mapping.

[42]  Lars Muckli,et al.  Cortical Plasticity of Audio–Visual Object Representations , 2008, Cerebral cortex.

[43]  G. Josse,et al.  The Brain’s Dorsal Route for Speech Represents Word Meaning: Evidence from Gesture , 2012, PloS one.

[44]  L. Marstaller,et al.  The multisensory perception of co-speech gestures – A review and meta-analysis of neuroimaging studies , 2014, Journal of Neurolinguistics.

[45]  Roel M. Willems,et al.  Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language , 2009, NeuroImage.

[46]  Marko Wilke,et al.  LI-tool: A new toolbox to assess lateralization in functional MR-data , 2007, Journal of Neuroscience Methods.

[47]  Anjali Raja Beharelle,et al.  Frontal and temporal contributions to understanding the iconic co‐speech gestures that accompany speech , 2014, Human brain mapping.

[48]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[49]  Tilo Kircher,et al.  Neural integration of speech and gesture in schizophrenia: Evidence for differential processing of metaphoric gestures , 2013, Human brain mapping.

[50]  C. Rorden,et al.  Stereotaxic display of brain lesions. , 2000, Behavioural neurology.

[51]  Bjørn Sætrevik,et al.  Cognitive conflict and inhibition in primed dichotic listening , 2009, Brain and Cognition.

[52]  Timothy O. Laumann,et al.  Methods to detect, characterize, and remove motion artifact in resting state fMRI , 2014, NeuroImage.

[53]  Kevin J. Riggs,et al.  Transcranial Magnetic Stimulation over Left Inferior Frontal and Posterior Temporal Cortex Disrupts Gesture-Speech Integration , 2018, The Journal of Neuroscience.

[54]  T. Kircher,et al.  Superior temporal sulcus disconnectivity during processing of metaphoric gestures in schizophrenia. , 2014, Schizophrenia bulletin.

[55]  B. Straube,et al.  Modulating the assessment of semantic speech–gesture relatedness via transcranial direct current stimulation of the left frontal cortex , 2017, Brain Stimulation.

[56]  Stefan Knecht,et al.  The assessment of hemispheric lateralization in functional MRI—Robustness and reproducibility , 2006, NeuroImage.

[57]  Wes McKinney,et al.  Data Structures for Statistical Computing in Python , 2010, SciPy.

[58]  Emiliano Ricciardi,et al.  Decomposing metaphor processing at the cognitive and neural level through functional magnetic resonance imaging , 2011, Brain Research Bulletin.

[59]  A. Jansen,et al.  Neural integration of iconic and unrelated coverbal gestures: A functional MRI study , 2009, Human brain mapping.

[60]  Jonas Obleser,et al.  Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions , 2010, NeuroImage.

[61]  Arne Nagels,et al.  The role of semantic abstractness and perceptual category in processing speech accompanied by gestures , 2013, Front. Behav. Neurosci..

[62]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[63]  A. Jacobs,et al.  Looking at the brains behind figurative language—A quantitative meta-analysis of neuroimaging studies on metaphor, idiom, and irony processing , 2012, Neuropsychologia.

[64]  Tilo Kircher,et al.  Memory Effects of Speech and Gesture Binding: Cortical and Hippocampal Activation in Relation to Subsequent Memory Performance , 2009, Journal of Cognitive Neuroscience.

[65]  P. Hagoort.  On Broca, brain, and binding: a new framework , 2005, Trends in Cognitive Sciences.

[66]  Stefan Kopp,et al.  Gesture and speech in interaction: An overview , 2014, Speech Commun..

[67]  A. Amedi,et al.  Functional imaging of human crossmodal identification and object recognition , 2005, Experimental Brain Research.

[68]  Cameron S Carter,et al.  Stimulus and response conflict processing during perceptual decision making , 2009, Cognitive, affective & behavioral neuroscience.

[69]  Michele T. Diaz,et al.  The influence of sentence novelty and figurativeness on brain activity , 2011, Neuropsychologia.

[70]  Abraham Z. Snyder,et al.  Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion , 2012, NeuroImage.

[71]  Elia Formisano,et al.  Hemispheric Differences in the Voluntary Control of Spatial Attention: Direct Evidence for a Right-Hemispheric Dominance within Frontal Cortex , 2013, Journal of Cognitive Neuroscience.

[72]  J. Kaiser,et al.  Object Familiarity and Semantic Congruency Modulate Responses in Cortical Audiovisual Integration Areas , 2007, The Journal of Neuroscience.

[73]  Vincent Schmithorst,et al.  A combined bootstrap/histogram analysis approach for computing a lateralization index from neuroimaging data , 2006, NeuroImage.

[74]  B. Argall,et al.  Unraveling multisensory integration: patchy organization within human STS multisensory cortex , 2004, Nature Neuroscience.

[75]  T. Kircher,et al.  The differentiation of iconic and metaphoric gestures: Common and unique integration processes , 2011, Human brain mapping.

[76]  Michael W. Cole,et al.  Conflict detection and resolution rely on a combination of common and distinct cognitive control networks , 2017, Neuroscience & Biobehavioral Reviews.

[77]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[78]  Edmund T. Rolls,et al.  Implementation of a new parcellation of the orbitofrontal cortex in the automated anatomical labeling atlas , 2015, NeuroImage.

[79]  Carlo Adolfo Porro,et al.  Neural substrates for observing and imagining non-object-directed actions , 2008, Social neuroscience.

[80]  S. Small,et al.  Gesture’s Neural Language , 2012, Front. Psychology.

[81]  C. Price.  The anatomy of language: contributions from functional neuroimaging , 2000, Journal of anatomy.