The effect of speech–gesture asynchrony on the neural coupling of interlocutors in interpreter-mediated communication

Abstract In everyday face-to-face communication, speakers use speech to transfer information and rely on co-occurring nonverbal cues, such as hand and facial gestures. The integration of speech and gestures facilitates both language comprehension and the skill of the theory of mind. Consecutive dialogue interpreting (DI) allows dyads of different linguistic backgrounds to communicate with each other. The interpreter interprets after the interlocutor has finished a turn, so the interlocutor watches the gesture first and hears the target language a few seconds later, resulting in speech–gesture asynchrony. In this study, we used the functional near-infrared spectroscopy hyperscanning technique to investigate the influence of speech–gesture asynchrony on different levels of communication. Twenty groups were recruited for the DI experiments. The results showed that when the interpreter performed consecutive interpreting, the time-lagged neural coupling at the temporoparietal junction decreased compared to simultaneous interpreting. It suggests that speech–gesture asynchrony significantly weakened the ability of interlocutors to understand each other’s mental state, and the decreased neural coupling was significantly correlated with the interpreter’s interpretation skill. In addition, the time-aligned neural coupling at the left inferior frontal gyrus increased, which suggests that, as compensation, the interlocutor’s verbal working memory increases in line with the communication process.

[1]  Xiaojun Cheng,et al.  Distinct neural couplings to shared goal and action coordination in joint action: evidence based on fNIRS hyperscanning , 2022, Social cognitive and affective neuroscience.

[2]  Chaozhe Zhu,et al.  NIRS-KIT: a MATLAB toolbox for both resting-state and task fNIRS data analysis , 2021, Neurophotonics.

[3]  Hui Zhao,et al.  Interpersonal Neural Synchronization during Interpersonal Touch Underlies Affiliative Pair Bonding between Romantic Couples. , 2020, Cerebral cortex.

[4]  Chunming Lu,et al.  A hierarchical model for interpersonal verbal communication , 2020, Social cognitive and affective neuroscience.

[5]  Yangmei Luo,et al.  Modality effects in verbal working memory updating: Transcranial direct current stimulation over human inferior frontal gyrus and posterior parietal cortex , 2020, Brain and Cognition.

[6]  Grace Y. Wang,et al.  What has social neuroscience learned from hyperscanning studies of spoken communication? A systematic review , 2020, Neuroscience & Biobehavioral Reviews.

[7]  Juliana E. Trach,et al.  Neural synchrony predicts children's learning of novel words , 2020, Cognition.

[8]  Taomei Guo,et al.  Affiliative bonding between teachers and students through interpersonal synchronisation in brain activity , 2020, Social cognitive and affective neuroscience.

[9]  Marcel Brass,et al.  Defining the neural correlates of spontaneous theory of mind (ToM): An fMRI multi-study investigation , 2019, NeuroImage.

[10]  Irene E. Harmsen Empathy in Autism Spectrum Disorder , 2019, Journal of Autism and Developmental Disorders.

[11]  Hui Zhao,et al.  Shared neural representations of syntax during online dyadic communication , 2019, NeuroImage.

[12]  E. Redcay,et al.  Using second-person neuroscience to elucidate the mechanisms of social interaction , 2019, Nature Reviews Neuroscience.

[13]  Chandan J. Vaidya,et al.  Temporal Derivative Distribution Repair (TDDR): A motion correction method for fNIRS , 2019, NeuroImage.

[14]  Elena Davitti Methodological explorations of interpreter-mediated interaction: novel insights from multimodal analysis , 2018, Qualitative Research.

[15]  J. Bavelas,et al.  Some pragmatic functions of conversational facial gestures1 , 2018, Gesture.

[16]  J. Hirsch,et al.  A cross-brain neural mechanism for human-to-human verbal communication , 2018, Social cognitive and affective neuroscience.

[17]  Zaizhu Han,et al.  Enhancement of teaching outcome through neural prediction of the students' knowledge state , 2018, Human brain mapping.

[18]  Li Liu,et al.  Neural mechanisms for selectively tuning in to the target speaker in a naturalistic noisy situation , 2018, Nature Communications.

[19]  Todd K. Moon,et al.  A functional near‐infrared spectroscopic investigation of speech production during reading , 2018, Human brain mapping.

[20]  Josef Perner,et al.  Specifying the brain anatomy underlying temporo‐parietal junction activations for theory of mind: A review using probabilistic atlases from different imaging modalities , 2017, Human brain mapping.

[21]  Takayuki Nozawa,et al.  Interpersonal frontopolar neural synchronization in group communication: An exploration toward fNIRS hyperscanning of natural interactions , 2016, NeuroImage.

[22]  T. Marinis,et al.  The Interface of Syntax with Pragmatics and Prosody in Children with Autism Spectrum Disorders , 2016, Journal of autism and developmental disorders.

[23]  Uri Hasson,et al.  Mirroring and beyond: coupled dynamics as a generalized framework for modelling social interactions , 2016, Philosophical Transactions of the Royal Society B: Biological Sciences.

[24]  Rebecca Saxe,et al.  Localizing Pain Matrix and Theory of Mind networks with both verbal and non-verbal stimuli , 2015, NeuroImage.

[25]  H. Wellman,et al.  Children’s belief- and desire-reasoning in the temporoparietal junction: evidence for specialization from functional near-infrared spectroscopy , 2015, Front. Hum. Neurosci..

[26]  Agustín Ibáñez,et al.  Two-Person Neuroscience and Naturalistic Social Communication: The Role of Language and Linguistic Variables in Brain-Coupling Research , 2014, Front. Psychiatry.

[27]  J. Bavelas,et al.  Including facial gestures in gesture-speech ensembles , 2014 .

[28]  Mandana Seyfeddinipur,et al.  From gesture in conversation to visible action as utterance , 2014 .

[29]  Chaozhe Zhu,et al.  Neural Synchronization during Face-to-Face Communication , 2012, The Journal of Neuroscience.

[30]  Claudio Baraldi,et al.  Coordinating Participation in Dialogue Interpreting , 2012 .

[31]  C. Honey,et al.  Not Lost in Translation: Neural Responses Shared Across Languages , 2012, The Journal of Neuroscience.

[32]  A. Fallgatter,et al.  Functional near-infrared spectroscopy for the assessment of speech related tasks , 2012, Brain and Language.

[33]  Xu Cui,et al.  NIRS-based hyperscanning reveals increased interpersonal coherence in superior frontal cortex during cooperation , 2012, NeuroImage.

[34]  Ivan Toni,et al.  Communicating without a functioning language system: Implications for the role of language in mentalizing , 2011, Neuropsychologia.

[35]  Zeshu Shao,et al.  The Role of Synchrony and Ambiguity in Speech–Gesture Integration during Comprehension , 2011, Journal of Cognitive Neuroscience.

[36]  Thomas C. Gunter,et al.  What Iconic Gesture Fragments Reveal about Gesture–Speech Integration: When Synchrony Is Lost, Memory Can Help , 2011, Journal of Cognitive Neuroscience.

[37]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[38]  Jakob Åsberg Patterns of language and discourse comprehension skills in school-aged children with autism spectrum disorders. , 2010, Scandinavian journal of psychology.

[39]  Luisa A. Ribeiro,et al.  Theory of mind and attentional bias to facial emotional expressions: a preliminary study. , 2010, Scandinavian journal of psychology.

[40]  U. Hasson,et al.  Speaker–listener neural coupling underlies successful communication , 2010, Proceedings of the National Academy of Sciences.

[41]  Yoko Hoshi,et al.  Functional near-infrared spectroscopy: current status and future prospects. , 2007, Journal of biomedical optics.

[42]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[43]  Kimberly J. Montgomery,et al.  Communicative hand gestures and object-directed hand movements activated the mirror neuron system. , 2007, Social cognitive and affective neuroscience.

[44]  Kevin J. Riggs,et al.  Is Belief Reasoning Automatic? , 2006, Psychological science.

[45]  C. Frith,et al.  The Neural Basis of Mentalizing , 2006, Neuron.

[46]  R. Saxe,et al.  Making sense of another mind: The role of the right temporo-parietal junction , 2005, Neuropsychologia.

[47]  Aslak Grinsted,et al.  Nonlinear Processes in Geophysics Application of the Cross Wavelet Transform and Wavelet Coherence to Geophysical Time Series , 2022 .

[48]  A. Kendon Gesture: Visible Action as Utterance , 2004 .

[49]  Hoi-Chung Leung,et al.  Frontal activations associated with accessing and evaluating information in working memory: an fMRI study , 2003, NeuroImage.

[50]  Mukesh Dhamala,et al.  Hyperscanning : Simultaneous fMRI during Linked Social Interactions , 2001 .

[51]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[52]  Hannu J. Aronen,et al.  Working Memory of Identification of Emotional Vocal Expressions: An fMRI Study , 2001, NeuroImage.

[53]  Edward T. Bullmore,et al.  Prolonged Reaction Time to a Verbal Working Memory Task Predicts Increased Power of Posterior Parietal Cortical Activation , 2000, NeuroImage.

[54]  S. Baron-Cohen,et al.  Another advanced test of theory of mind: evidence from very high functioning adults with autism or asperger syndrome. , 1997, Journal of child psychology and psychiatry, and allied disciplines.

[55]  Justine Cassell,et al.  Communicative Effects of Speech-Mismatched Gestures , 1994 .

[56]  M. Studdert-Kennedy Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[57]  S. Baron-Cohen,et al.  Does the autistic child have a “theory of mind” ? , 1985, Cognition.

[58]  D. Premack,et al.  Does the chimpanzee have a theory of mind? , 1978, Behavioral and Brain Sciences.

[59]  F. Jöbsis Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. , 1977, Science.

[60]  Annalisa Sandrelli Simultaneous dialogue interpreting: Coordinating interaction in interpreter-mediated football press conferences , 2017 .

[61]  Sang-Bin Lee Developing an analytic scale for assessing undergraduate students’ consecutive interpreting performances , 2015 .

[62]  John A. Daly,et al.  The Sage handbook of interpersonal communication , 2011 .

[63]  H. Bot Dialogue interpreting as a specific case of reported speech , 2005 .

[64]  C. Torrence,et al.  A Practical Guide to Wavelet Analysis. , 1998 .