The neural basis of hand gesture comprehension: A meta-analysis of functional magnetic resonance imaging studies

Gestures play an important role in face-to-face communication and have been increasingly studied via functional magnetic resonance imaging. Although a large amount of data has been provided to describe the neural substrates of gesture comprehension, these findings have never been quantitatively summarized and the conclusion is still unclear. This activation likelihood estimation meta-analysis investigated the brain networks underpinning gesture comprehension while considering the impact of gesture type (co-speech gestures vs. speech-independent gestures) and task demand (implicit vs. explicit) on the brain activation of gesture comprehension. The meta-analysis of 31 papers showed that as hand actions, gestures involve a perceptual-motor network important for action recognition. As meaningful symbols, gestures involve a semantic network for conceptual processing. Finally, during face-to-face interactions, gestures involve a network for social emotive processes. Our finding also indicated that gesture type and task demand influence the involvement of the brain networks during gesture comprehension. The results highlight the complexity of gesture comprehension, and suggest that future research is necessary to clarify the dynamic interactions among these networks.

[1]  Andreas Hennenlotter,et al.  Neural correlates of the processing of co-speech gestures , 2008, NeuroImage.

[2]  D. Callan,et al.  Altered integration of speech and gesture in children with autism spectrum disorders , 2012, Brain and behavior.

[3]  Randy L Buckner,et al.  Common and dissociable activation patterns associated with controlled semantic and phonological processing: evidence from FMRI adaptation. , 2005, Cerebral cortex.

[4]  Steven L. Small,et al.  Neural development of networks for audiovisual speech comprehension , 2010, Brain and Language.

[5]  K. Zilles,et al.  Coordinate‐based activation likelihood estimation meta‐analysis of neuroimaging data: A random‐effects approach based on empirical estimates of spatial uncertainty , 2009, Human brain mapping.

[6]  Jeremy I. Skipper,et al.  Co‐speech gestures influence neural activity in brain regions associated with processing semantic information , 2009, Human brain mapping.

[7]  J. Tanji,et al.  Distinctions between dorsal and ventral premotor areas: anatomical connectivity and functional properties , 2007, Current Opinion in Neurobiology.

[8]  A A Montgomery,et al.  Auditory and visual contributions to the perception of consonants. , 1974, Journal of speech and hearing research.

[9]  Anjali Raja Beharelle,et al.  Frontal and temporal contributions to understanding the iconic co‐speech gestures that accompany speech , 2014, Human brain mapping.

[10]  J. Haxby,et al.  Distinct representations of eye gaze and identity in the distributed human neural system for face perception , 2000, Nature Neuroscience.

[11]  Natalie M. Myres,et al.  Distinctive Paleo-Indian Migration Routes from Beringia Marked by Two Rare mtDNA Haplogroups , 2009, Current Biology.

[12]  J. Mazziotta,et al.  Cortical mechanisms of human imitation. , 1999, Science.

[13]  D. Heeger,et al.  In this issue , 2002, Nature Reviews Drug Discovery.

[14]  A. Young,et al.  Impaired recognition and experience of disgust following brain injury , 2000, Nature Neuroscience.

[15]  Steven L. Small,et al.  Abstract Coding of Audiovisual Speech: Beyond Sensory Representation , 2007, Neuron.

[16]  Lee M. Miller,et al.  Behavioral/systems/cognitive Perceptual Fusion and Stimulus Coincidence in the Cross- Modal Integration of Speech , 2022 .

[17]  Benjamin T. Files,et al.  The visual mismatch negativity elicited with visual speech stimuli , 2013, Front. Hum. Neurosci..

[18]  Sook-Lei Liew,et al.  Familiarity modulates mirror neuron and mentalizing regions during intention understanding , 2011, Human brain mapping.

[19]  Mikko Sams,et al.  Processing of audiovisual speech in Broca's area , 2005, NeuroImage.

[20]  Susan Goldin-Meadow.  Le rôle des gestes dans la création et l’acquisition du langage , 2010, Enfance.

[21]  F. Binkofski,et al.  Motor functions of the Broca’s region , 2004, Brain and Language.

[22]  Jordan Grafman,et al.  Observing social gestures: an fMRI study , 2008, Experimental Brain Research.

[23]  N. Sadato,et al.  Neural networks for action representation: a functional magnetic-resonance imaging and dynamic causal modeling study , 2012, Front. Hum. Neurosci..

[24]  S. Petersen,et al.  PET activation of posterior temporal regions during auditory word presentation and verb generation. , 1996, Cerebral cortex.

[25]  Charles D. Smith,et al.  Dissociation of Automatic and Strategic Lexical-Semantics: Functional Magnetic Resonance Imaging Evidence for Differing Roles of Multiple Frontotemporal Regions , 2006, The Journal of Neuroscience.

[26]  A. Damasio,et al.  Subcortical and cortical brain activity during the feeling of self-generated emotions , 2000, Nature Neuroscience.

[27]  Andrew N. Meltzoff,et al.  Neural circuits involved in imitation and perspective-taking , 2006, NeuroImage.

[28]  A. Macleod,et al.  Quantifying the contribution of vision to speech perception in noise. , 1987, British journal of audiology.

[29]  Markus Junghöfer,et al.  Neural systems of visual attention responding to emotional gestures , 2009, NeuroImage.

[30]  S. Bookheimer,et al.  Form and Content Dissociating Syntax and Semantics in Sentence Comprehension , 1999, Neuron.

[31]  Tilo Kircher,et al.  Social cues, mentalizing and the neural processing of speech accompanied by gestures , 2010, Neuropsychologia.

[32]  Ehud Zohary,et al.  Dissociation between Ventral and Dorsal fMRI Activation during Object and Action Recognition , 2005, Neuron.

[33]  Bernard Mazoyer,et al.  Meta-analyzing left hemisphere language areas: Phonology, semantics, and sentence processing , 2006, NeuroImage.

[34]  R. E Passingham,et al.  Activations related to “mirror” and “canonical” neurones in the human brain: an fMRI study , 2003, NeuroImage.

[35]  Aina Puce,et al.  Electrophysiology and brain imaging of biological motion. , 2003, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[36]  Stephen M. Rao,et al.  Human Brain Language Areas Identified by Functional Magnetic Resonance Imaging , 1997, The Journal of Neuroscience.

[37]  K. Willmes,et al.  Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures , 2009, Neuropsychologia.

[38]  C. Frith,et al.  Dissociable neural pathways for the perception and recognition of expressive and instrumental gestures , 2004, Neuropsychologia.

[39]  M. Erb,et al.  Brain activity underlying emotional valence and arousal: A response‐related fMRI study , 2004, Human brain mapping.

[40]  Jeremy I. Skipper,et al.  Speech-associated gestures, Broca’s area, and the human mirror system , 2007, Brain and Language.

[41]  Cinzia Di Dio,et al.  The neural correlates of velocity processing during the observation of a biological effector in the parietal and premotor cortex , 2013, NeuroImage.

[42]  A. Jansen,et al.  Neural integration of iconic and unrelated coverbal gestures: A functional MRI study , 2009, Human brain mapping.

[43]  Julie Grèzes,et al.  Look at me, I'll remember you , 2012, Human brain mapping.

[44]  N. A. Borghese,et al.  Different Brain Correlates for Watching Real and Virtual Hand Actions , 2001, NeuroImage.

[45]  E. Liebenthal,et al.  Neural pathways for visual speech perception , 2014, Front. Neurosci..

[46]  G. Fink,et al.  Neurofunctional modulation of brain regions by distinct forms of motor cognition and movement features , 2009, Human brain mapping.

[47]  R. J. Seitz,et al.  The neural correlates of affect reading: An fMRI study on faces and gestures , 2013, Behavioural Brain Research.

[48]  Wolfgang Grodd,et al.  Identification of emotional intonation evaluated by fMRI , 2005, NeuroImage.

[49]  Jonas Obleser,et al.  Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions , 2010, NeuroImage.

[50]  Paul B. Johnson,et al.  Cortical networks for visual reaching: physiological and anatomical organization of frontal and parietal lobe arm regions. , 1996, Cerebral cortex.

[51]  G. Vingerhoets.  Contribution of the posterior parietal cortex in reaching, grasping, and using objects and tools , 2014, Front. Psychol..

[52]  A. Meltzoff,et al.  A PET Exploration of the Neural Mechanisms Involved in Reciprocal Imitation , 2002, NeuroImage.

[53]  M. Alibali,et al.  Gesture's role in speaking, learning, and creating language. , 2013, Annual review of psychology.

[54]  Steven L. Small,et al.  A Network Model of Observation and Imitation of Speech , 2012, Front. Psychology.

[55]  N. Sadato,et al.  The Brain Network Underlying the Recognition of Hand Gestures in the Blind: The Supramodal Role of the Extrastriate Body Area , 2014, The Journal of Neuroscience.

[56]  G. Rizzolatti,et al.  The functional role of the parieto-frontal mirror circuit: interpretations and misinterpretations , 2010, Nature Reviews Neuroscience.

[57]  D. Hu,et al.  Neurobiological basis of head motion in brain imaging , 2014, Proceedings of the National Academy of Sciences.

[58]  Kimberly J. Montgomery,et al.  Communicative hand gestures and object-directed hand movements activated the mirror neuron system. , 2007, Social cognitive and affective neuroscience.

[59]  Shigeru Sato,et al.  Cortical Mechanisms Involved in the Processing of Verbs: An fMRI Study , 2006, Journal of Cognitive Neuroscience.

[60]  W. Levelt,et al.  The spatial and temporal signatures of word production components , 2004, Cognition.

[61]  T. Kircher,et al.  Semantic relation vs. surprise: The differential effects of related and unrelated co-verbal gestures on neural encoding and subsequent recognition , 2014, Brain Research.

[62]  Susan Goldin-Meadow,et al.  Gesture in the developing brain. , 2012, Developmental science.

[63]  R. Veit,et al.  Differential cerebral activation during observation of expressive gestures and motor acts , 2006, Neuropsychologia.

[64]  G. Rizzolatti,et al.  Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study , 2001, The European journal of neuroscience.

[65]  T. Kircher,et al.  The differentiation of iconic and metaphoric gestures: Common and unique integration processes , 2011, Human brain mapping.

[66]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[67]  D Y von Cramon,et al.  Segregating semantic and syntactic aspects of processing in the human brain: an fMRI investigation of different word types. , 2000, Cerebral cortex.

[68]  Dieter Vaitl,et al.  Differential activation of the lateral premotor cortex during action observation , 2010, BMC Neuroscience.

[69]  S. Jackson,et al.  Parietal cortex coding of limb posture: In search of the body-schema , 2010, Neuropsychologia.

[70]  J. Mattingley,et al.  Brain regions with mirror properties: A meta-analysis of 125 human fMRI studies , 2012, Neuroscience & Biobehavioral Reviews.

[71]  P. Matthews,et al.  Semantic Processing in the Left Inferior Prefrontal Cortex: A Combined Functional Magnetic Resonance Imaging and Transcranial Magnetic Stimulation Study , 2003, Journal of Cognitive Neuroscience.

[72]  B. Mensour,et al.  Neural correlates of sad feelings in healthy girls , 2003, Neuroscience.

[73]  Jeremy I. Skipper,et al.  Gestures Orchestrate Brain Networks for Language Understanding , 2009, Current Biology.

[74]  M. Sams,et al.  Attention to visual speech gestures enhances hemodynamic activity in the left planum temporale , 2006, Human brain mapping.

[75]  S. Aglioti,et al.  Neural Underpinnings of Gesture Discrimination in Patients with Limb Apraxia , 2008, The Journal of Neuroscience.

[76]  Arne Nagels,et al.  Supramodal neural processing of abstract information conveyed by speech and gesture , 2013, Front. Behav. Neurosci..

[77]  M. Gentilucci,et al.  The eye contact effect in request and emblematic hand gestures , 2014, The European journal of neuroscience.

[78]  Mirta F. Villarreal,et al.  The Effect of the Visual Context in the Recognition of Symbolic Gestures , 2012, PloS one.

[79]  A. Braun,et al.  Symbolic gestures and spoken language are processed by a common neural system , 2009, Proceedings of the National Academy of Sciences.

[80]  D. Perani,et al.  The neural correlates of verb and noun processing. A PET study. , 1999, Brain : a journal of neurology.

[81]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.

[82]  Roel M. Willems,et al.  Neural evidence for the interplay between language, gesture, and action: A review , 2007, Brain and Language.

[83]  G. Rizzolatti,et al.  Brain function overlaps when people observe emblems, speech, and grasping , 2013, Neuropsychologia.

[84]  Simon B. Eickhoff,et al.  Meta-Analyses in Basic and Clinical Neuroscience: State of the Art and Perspective , 2013 .

[85]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[86]  James V. Haxby,et al.  Mirror Neuron System Differentially Activated by Facial Expressions and Social Hand Gestures: A Functional Magnetic Resonance Imaging Study , 2008, Journal of Cognitive Neuroscience.

[87]  Tilo Kircher,et al.  A Supramodal Neural Network for Speech and Gesture Semantics: An fMRI Study , 2012, PloS one.

[88]  G. Rizzolatti,et al.  Mirror neurons and mirror systems in monkeys and humans. , 2008, Physiology.

[89]  Leslie G. Ungerleider,et al.  Discrete Cortical Regions Associated with Knowledge of Color and Knowledge of Action , 1995, Science.

[90]  Marianne Patt,et al.  Reduced Ventrolateral fMRI Response during Observation of Emotional Gestures Related to the Degree of Dopaminergic Impairment in Parkinson Disease , 2009, Journal of Cognitive Neuroscience.

[91]  G. Josse,et al.  The Brain’s Dorsal Route for Speech Represents Word Meaning: Evidence from Gesture , 2012, PloS one.

[92]  L. Bernstein,et al.  Quantified acoustic–optical speech signal incongruity identifies cortical sites of audiovisual speech processing , 2008, Brain Research.

[93]  F. Binkofski,et al.  The mirror neuron system and action recognition , 2004, Brain and Language.

[94]  S. Thompson-Schill.  Neuroimaging studies of semantic memory: inferring “how” from “where” , 2003, Neuropsychologia.

[95]  M. Corballis,et al.  From manual gesture to speech: A gradual transition , 2006, Neuroscience & Biobehavioral Reviews.

[96]  Rainer Goebel,et al.  Playing Charades in the fMRI: Are Mirror and/or Mentalizing Areas Involved in Gestural Communication? , 2009, PloS one.

[97]  Bruno G. Bara,et al.  Intention Processing in Communication: A Common Brain Network for Language and Gestures , 2011, Journal of Cognitive Neuroscience.

[98]  J. Haxby,et al.  Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects , 1999, Nature Neuroscience.

[99]  B. Mensour,et al.  Neural basis of emotional self-regulation in childhood , 2004, Neuroscience.

[100]  Guinevere F. Eden,et al.  Meta-Analysis of the Functional Neuroanatomy of Single-Word Reading: Method and Validation , 2002, NeuroImage.

[101]  G. Rizzolatti.  The mirror neuron system and its function in humans , 2005, Anatomy and Embryology.

[102]  G. Rizzolatti,et al.  Parietal Lobe: From Action Organization to Intention Understanding , 2005, Science.

[103]  S. Small,et al.  Gesture’s Neural Language , 2012, Front. Psychology.

[104]  R. Elliott,et al.  Dissociable functions in the medial and lateral orbitofrontal cortex: evidence from human neuroimaging studies. , 2000, Cerebral cortex.

[105]  Steven L. Small,et al.  Listening to talking faces: motor cortical activation during speech perception , 2005, NeuroImage.

[106]  Michael J. Martinez,et al.  Bias between MNI and Talairach coordinates analyzed using the ICBM‐152 brain template , 2007, Human brain mapping.

[107]  Allan L. Reiss,et al.  Revealing the neural networks associated with processing of natural social interaction and the related effects of actor-orientation and face-visibility , 2014, NeuroImage.

[108]  Ivan Toni,et al.  Eye'm talking to you: speakers' gaze direction modulates co-speech gesture processing in the right MTG. , 2015, Social cognitive and affective neuroscience.

[109]  P. Jackson,et al.  The neural network of motor imagery: An ALE meta-analysis , 2013, Neuroscience & Biobehavioral Reviews.

[110]  Gregory McCarthy,et al.  Polysensory interactions along lateral temporal regions evoked by audiovisual speech. , 2003, Cerebral cortex.

[111]  D. Callan,et al.  Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception , 2009, Human brain mapping.

[112]  Stefan Heim,et al.  Differential role of the Mentalizing and the Mirror Neuron system in the imitation of communicative gestures , 2013, NeuroImage.

[113]  G. Rizzolatti,et al.  Understanding motor events: a neurophysiological study , 2004, Experimental Brain Research.

[114]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[115]  Arne Nagels,et al.  The role of semantic abstractness and perceptual category in processing speech accompanied by gestures , 2013, Front. Behav. Neurosci..

[116]  Olav Jansen,et al.  fMRI: Basics and Clinical Applications , 2013 .

[117]  Mikko Sams,et al.  Mental Action Simulation Synchronizes Action–Observation Circuits across Individuals , 2014, The Journal of Neuroscience.

[118]  S. Goldin‐Meadow.  GESTURE'S ROLE IN CREATING AND LEARNING LANGUAGE. , 2010, Enfance; psychologie, pedagogie, neuropsychiatrie, sociologie.

[119]  Dimitrios Pantazis,et al.  Visual phonetic processing localized using speech and nonspeech face gestures in video and point‐light displays , 2011, Human brain mapping.

[120]  Leonardo Fogassi,et al.  Motor functions of the parietal lobe , 2005, Current Opinion in Neurobiology.

[121]  Anjan Chatterjee,et al.  Action Concepts in the Brain: An Activation Likelihood Estimation Meta-analysis , 2013, Journal of Cognitive Neuroscience.

[122]  E. Rolls,et al.  Face and voice expression identification in patients with emotional and behavioural changes following ventral frontal lobe damage , 1996, Neuropsychologia.

[123]  Diane M. Beck,et al.  Pulsed Out of Awareness: EEG Alpha Oscillations Represent a Pulsed-Inhibition of Ongoing Cortical Processing , 2011, Front. Psychology.

[124]  Robert Lindenberg,et al.  Communication with emblematic gestures: Shared and distinct neural correlates of expression and reception , 2012, Human brain mapping.

[125]  Sakiko Yoshikawa,et al.  Commonalities in the neural mechanisms underlying automatic attentional shifts by gaze, gestures, and symbols , 2009, NeuroImage.

[126]  R. Leiguarda,et al.  The neural substrate of gesture recognition , 2008, Neuropsychologia.

[127]  G. Rizzolatti,et al.  Neurophysiological mechanisms underlying the understanding and imitation of action , 2001, Nature Reviews Neuroscience.

[128]  Roel M. Willems,et al.  Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language , 2009, NeuroImage.

[129]  Peter A. Bandettini,et al.  Separating respiratory-variation-related fluctuations from neuronal-activity-related fluctuations in fMRI , 2006, NeuroImage.

[130]  Laurel J Buxbaum,et al.  Critical brain regions for action recognition: lesion symptom mapping in left hemisphere stroke. , 2010, Brain : a journal of neurology.