Perceived communicative intent in gesture and language modulates the superior temporal sulcus

Behavioral evidence and theory suggest gesture and language processing may be part of a shared cognitive system for communication. While much research demonstrates both gesture and language recruit regions along perisylvian cortex, relatively less work has tested functional segregation within these regions on an individual level. Additionally, while most work has focused on a shared semantic network, less has examined shared regions for processing communicative intent. To address these questions, functional and structural MRI data were collected from 24 adult participants while viewing videos of an experimenter producing communicative, Participant‐Directed Gestures (PDG) (e.g., “Hello, come here”), noncommunicative Self‐adaptor Gestures (SG) (e.g., smoothing hair), and three written text conditions: (1) Participant‐Directed Sentences (PDS), matched in content to PDG, (2) Third‐person Sentences (3PS), describing a character's actions from a third‐person perspective, and (3) meaningless sentences, Jabberwocky (JW). Surface‐based conjunction and individual functional region of interest analyses identified shared neural activation between gesture (PDGvsSG) and language processing using two different language contrasts. Conjunction analyses of gesture (PDGvsSG) and Third‐person Sentences versus Jabberwocky revealed overlap within left anterior and posterior superior temporal sulcus (STS). Conjunction analyses of gesture and Participant‐Directed Sentences to Third‐person Sentences revealed regions sensitive to communicative intent, including the left middle and posterior STS and left inferior frontal gyrus. Further, parametric modulation using participants' ratings of stimuli revealed sensitivity of left posterior STS to individual perceptions of communicative intent in gesture. These data highlight an important role of the STS in processing participant‐directed communicative intent through gesture and language. Hum Brain Mapp 37:3444–3461, 2016. © 2016 Wiley Periodicals, Inc.

[1]  Anders M. Dale,et al.  An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest , 2006, NeuroImage.

[2]  A. Dale,et al.  High‐resolution intersubject averaging and a coordinate system for the cortical surface , 1999, Human brain mapping.

[3]  K. Pelphrey,et al.  An integrative neural model of social perception, action observation, and theory of mind , 2015, Neuroscience & Biobehavioral Reviews.

[4]  Tilo Kircher,et al.  A Supramodal Neural Network for Speech and Gesture Semantics: An fMRI Study , 2012, PloS one.

[5]  A. Koller,et al.  Speech Acts: An Essay in the Philosophy of Language , 1969 .

[6]  S. Small,et al.  Gesture’s Neural Language , 2012, Front. Psychology.

[7]  Arne Nagels,et al.  Supramodal neural processing of abstract information conveyed by speech and gesture , 2013, Front. Behav. Neurosci..

[8]  Susan Goldin-Meadow,et al.  Early gesture selectively predicts later language learning. , 2009, Developmental science.

[9]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[10]  M. Gentilucci,et al.  The eye contact effect in request and emblematic hand gestures , 2014, The European journal of neuroscience.

[11]  E. Redcay.  The superior temporal sulcus performs a common function for social and speech perception: Implications for the emergence of autism , 2008, Neuroscience & Biobehavioral Reviews.

[12]  M. Raichle,et al.  Searching for a baseline: Functional imaging and the resting human brain , 2001, Nature Reviews Neuroscience.

[13]  Cathy J. Price,et al.  Functional Heterogeneity within the Default Network during Semantic Processing and Speech Production , 2012, Front. Psychology.

[14]  付伶俐.  打磨Using Language,倡导新理念 , 2014 .

[15]  H. Walter,et al.  Do you mean me? Communicative intentions recruit the mirror and the mentalizing system. , 2014, Social cognitive and affective neuroscience.

[16]  Marisa O. Hollinshead,et al.  The organization of the human cerebral cortex estimated by intrinsic functional connectivity. , 2011, Journal of neurophysiology.

[17]  T. Kircher,et al.  The differentiation of iconic and metaphoric gestures: Common and unique integration processes , 2011, Human brain mapping.

[18]  R. N. Spreng,et al.  I remember you: A role for memory in social cognition and the functional neuroanatomy of their interaction , 2012, Brain Research.

[19]  Nancy Kanwisher,et al.  Syntactic processing in the human brain: What we know, what we don’t know, and a suggestion for how to proceed , 2011, Brain and Language.

[20]  Ivan Toni,et al.  Eye'm talking to you: speakers' gaze direction modulates co-speech gesture processing in the right MTG. , 2015, Social cognitive and affective neuroscience.

[21]  Maximilian Riesenhuber,et al.  Individual Variability in Location Impacts Orthographic Selectivity in the “Visual Word Form Area” , 2013, The Journal of Neuroscience.

[22]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[23]  Elizabeth Redcay,et al.  Interaction matters: A perceived social partner alters the neural processing of human speech , 2016, NeuroImage.

[24]  D. Callan,et al.  Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception , 2009, Human brain mapping.

[25]  S. Levinson,et al.  Brain Mechanisms Underlying Human Communication , 2009, Front. Hum. Neurosci..

[26]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[27]  Jonas Obleser,et al.  Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions , 2010, NeuroImage.

[28]  R. Mar.  The neural bases of social cognition and story comprehension. , 2011, Annual review of psychology.

[29]  David Gow,et al.  Effects of tasks on BOLD signal responses to sentence contrasts: Review and commentary , 2012, Brain and Language.

[30]  C. Price.  The anatomy of language: a review of 100 fMRI studies published in 2009 , 2010, Annals of the New York Academy of Sciences.

[31]  Nancy Kanwisher,et al.  Functional Organization of Social Perception and Cognition in the Superior Temporal Sulcus , 2015, Cerebral cortex.

[32]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[33]  R. Saxe,et al.  Look at this: the neural correlates of initiating and responding to bids for joint attention , 2012, Front. Hum. Neurosci..

[34]  K. Willmes,et al.  Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures , 2009, Neuropsychologia.

[35]  Jesper Andersson,et al.  Valid conjunction inference with the minimum statistic , 2005, NeuroImage.

[36]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[37]  Anjali Raja Beharelle,et al.  Frontal and temporal contributions to understanding the iconic co‐speech gestures that accompany speech , 2014, Human brain mapping.

[38]  G. Fink,et al.  Being with virtual others: Neural correlates of social interaction , 2006, Neuropsychologia.

[39]  Kristian Tylén,et al.  Interaction vs. observation: distinctive modes of social cognition in human brain and behavior? A combined fMRI and eye-tracking study , 2012, Front. Hum. Neurosci..

[40]  S. Goldin-Meadow,et al.  Differences in Early Gesture Explain SES Disparities in Child Vocabulary Size at School Entry , 2009, Science.

[41]  Susan Goldin-Meadow,et al.  Gesture in the developing brain. , 2012, Developmental science.

[42]  Tilo Kircher,et al.  Social cues, mentalizing and the neural processing of speech accompanied by gestures , 2010, Neuropsychologia.

[43]  Dare A. Baldwin,et al.  Infants' contribution to the achievement of joint reference. , 1991, Child development.

[44]  S. Dehaene,et al.  Cortical representation of the constituent structure of sentences , 2011, Proceedings of the National Academy of Sciences.

[45]  Paul E. Downing,et al.  A comparison of volume-based and surface-based multi-voxel pattern analysis , 2011, NeuroImage.

[46]  Rebecca Saxe,et al.  Live face-to-face interaction during fMRI: A new tool for social cognitive neuroscience , 2010, NeuroImage.

[47]  Andrew N Meltzoff,et al.  Infant gaze following and pointing predict accelerated vocabulary growth through two years of age: a longitudinal, growth curve modeling study* , 2008, Journal of Child Language.

[48]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[49]  J. Binder,et al.  A Parametric Manipulation of Factors Affecting Task-induced Deactivation in Functional Neuroimaging , 2003, Journal of Cognitive Neuroscience.

[50]  D. Saucier,et al.  Enriched childhood experiences moderate age-related motor and cognitive decline , 2012, Front. Behav. Neurosci..

[51]  A. Braun,et al.  Symbolic gestures and spoken language are processed by a common neural system , 2009, Proceedings of the National Academy of Sciences.

[52]  A. Dick,et al.  The Neurobiology of Gesture and Its Development , 2016 .

[53]  Andreas Hennenlotter,et al.  Neural correlates of the processing of co-speech gestures , 2008, NeuroImage.

[54]  S. Goldin‐Meadow.  The development of gesture and speech as an integrated system. , 1998, New directions for child development.

[55]  L. J. Chapman,et al.  The measurement of handedness , 1987, Brain and Cognition.

[56]  D. Sperber,et al.  Relevance: Communication and Cognition , 1989 .

[57]  Elizabeth Redcay,et al.  Communicative Signals Promote Object Recognition Memory and Modulate the Right Posterior STS , 2016, Journal of Cognitive Neuroscience.

[58]  A M Dale,et al.  Measuring the thickness of the human cerebral cortex from magnetic resonance images. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[59]  Brenna Argall,et al.  SUMA: an interface for surface-based intra- and inter-subject analysis with AFNI , 2004, 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro (IEEE Cat No. 04EX821).

[60]  D. Callan,et al.  Altered integration of speech and gesture in children with autism spectrum disorders , 2012, Brain and behavior.

[61]  Nathan Caruana,et al.  A frontotemporoparietal network common to initiating and responding to joint attention bids , 2014, NeuroImage.

[62]  G. McCarthy,et al.  Functional Heterogeneity and Convergence in the Right Temporoparietal Junction. , 2016, Cerebral cortex.

[63]  Jeremy I. Skipper,et al.  Speech-associated gestures, Broca’s area, and the human mirror system , 2007, Brain and Language.

[64]  HeinGrit,et al.  Superior temporal sulcus---it's my area , 2008 .

[65]  F. Dick,et al.  Language, gesture, and the developing brain. , 2002, Developmental psychobiology.

[66]  Arne Nagels,et al.  Feeling addressed! The role of body orientation and co‐speech gesture in social communication , 2015, Human brain mapping.

[67]  Roel M. Willems,et al.  Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language , 2009, NeuroImage.

[68]  Rutvik H. Desai,et al.  The neurobiology of semantic memory , 2011, Trends in Cognitive Sciences.

[69]  Nancy Kanwisher,et al.  Functional specificity for high-level linguistic processing in the human brain , 2011, Proceedings of the National Academy of Sciences.

[70]  Marc Brysbaert,et al.  Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English , 2009, Behavior research methods.

[71]  G. Rizzolatti,et al.  Brain function overlaps when people observe emblems, speech, and grasping , 2013, Neuropsychologia.

[72]  Jeremy I. Skipper,et al.  Co‐speech gestures influence neural activity in brain regions associated with processing semantic information , 2009, Human brain mapping.

[73]  Richard C. Reynolds,et al.  SUMA , 2012, NeuroImage.

[74]  Mark D'Esposito,et al.  The Prefrontal Cortex Modulates Category Selectivity in Human Extrastriate Cortex , 2011, Journal of Cognitive Neuroscience.

[75]  A. Braun,et al.  Neural responses to meaningless pseudosigns: Evidence for sign-based phonetic processing in superior temporal cortex , 2011, Brain and Language.

[76]  F. Dick,et al.  Beyond Phrenology: Brain and Language in the Next Millennium , 2000, Brain and Language.

[77]  Jennifer C. Britton,et al.  Linear mixed-effects modeling approach to FMRI group analysis , 2013, NeuroImage.

[78]  K. Vogeley,et al.  Toward a second-person neuroscience 1 , 2013, Behavioral and Brain Sciences.

[79]  M. Tomasello,et al.  Understanding and sharing intentions: The origins of cultural cognition , 2005, Behavioral and Brain Sciences.

[80]  Bruno G. Bara,et al.  Intention Processing in Communication: A Common Brain Network for Language and Gestures , 2011, Journal of Cognitive Neuroscience.

[81]  Allen Braun,et al.  Neural correlates of human action observation in hearing and deaf subjects , 2007, Brain Research.

[82]  A. Dale,et al.  Cortical Surface-Based Analysis II: Inflation, Flattening, and a Surface-Based Coordinate System , 1999, NeuroImage.

[83]  Jae-Hun Kim,et al.  Spatial accuracy of fMRI activation influenced by volume- and surface-based spatial smoothing techniques , 2007, NeuroImage.