Hearing and seeing meaning in noise: Alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension

During face‐to‐face communication, listeners integrate speech with gestures. The semantic information conveyed by iconic gestures (e.g., a drinking gesture) can aid speech comprehension in adverse listening conditions. In this magnetoencephalography (MEG) study, we investigated the spatiotemporal neural oscillatory activity associated with gestural enhancement of degraded speech comprehension. Participants watched videos of an actress uttering clear or degraded speech, accompanied by a gesture or not and completed a cued‐recall task after watching every video. When gestures semantically disambiguated degraded speech comprehension, an alpha and beta power suppression and a gamma power increase revealed engagement and active processing in the hand‐area of the motor cortex, the extended language network (LIFG/pSTS/STG/MTG), medial temporal lobe, and occipital regions. These observed low‐ and high‐frequency oscillatory modulations in these areas support general unification, integration and lexical access processes during online language comprehension, and simulation of and increased visual attention to manual gestures over time. All individual oscillatory power modulations associated with gestural enhancement of degraded speech comprehension predicted a listener's correct disambiguation of the degraded verb after watching the videos. Our results thus go beyond the previously proposed role of oscillatory dynamics in unimodal degraded speech comprehension and provide first evidence for the role of low‐ and high‐frequency oscillations in predicting the integration of auditory and visual information at a semantic level.

[1]  C. Eulitz,et al.  Top-down knowledge supports the retrieval of lexical information from degraded speech , 2007, Brain Research.

[2]  Björn Herrmann,et al.  Neural Alpha Dynamics in Younger and Older Listeners Reflect Acoustic Challenges and Predictive Benefits , 2015, The Journal of Neuroscience.

[3]  Jonas Obleser,et al.  Suppressed alpha oscillations predict intelligibility of speech and its acoustic details. , 2012, Cerebral cortex.

[4]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[5]  Salvador Soto-Faraco,et al.  Synchronization by the hand: the sight of gestures modulates low-frequency activity in brain responses to continuous speech , 2015, Front. Hum. Neurosci..

[6]  C. Frith,et al.  Relationship between Activity in Human Primary Motor Cortex during Action Observation and the Mirror Neuron System , 2009, PloS one.

[7]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[8]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.

[9]  Roel M. Willems,et al.  Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language , 2009, NeuroImage.

[10]  T. Kircher,et al.  The EEG and fMRI signatures of neural integration: An investigation of meaningful gestures and corresponding speech , 2015, Neuropsychologia.

[11]  Tilo Kircher,et al.  A Supramodal Neural Network for Speech and Gesture Semantics: An fMRI Study , 2012, PloS one.

[12]  A. Jansen,et al.  Neural integration of iconic and unrelated coverbal gestures: A functional MRI study , 2009, Human brain mapping.

[13]  Linda Drijvers,et al.  Visual Context Enhanced: The Joint Contribution of Iconic Gestures and Visible Speech to Degraded Speech Comprehension. , 2017, Journal of speech, language, and hearing research : JSLHR.

[14]  Nikos K. Logothetis,et al.  Desynchronization and rebound of beta oscillations during conscious and unconscious local neuronal processing in the macaque lateral prefrontal cortex , 2013, Front. Psychol..

[15]  Ellen F. Lau,et al.  A cortical network for semantics: (de)constructing the N400 , 2008, Nature Reviews Neuroscience.

[16]  M. Botvinick,et al.  Parsing executive processes: strategic vs. evaluative functions of the anterior cingulate cortex. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[17]  J. Kaiser,et al.  Human gamma-frequency oscillations associated with attention and memory , 2007, Trends in Neurosciences.

[18]  W. Klimesch,et al.  EEG alpha oscillations: The inhibition–timing hypothesis , 2007, Brain Research Reviews.

[19]  Thomas R Knösche,et al.  Tangential derivative mapping of axial MEG applied to event-related desynchronization research , 2000, Clinical Neurophysiology.

[20]  Rolf A. Zwaan,et al.  The functional role of motor activation in language processing: Motor cortical oscillations support lexical-semantic retrieval , 2010, NeuroImage.

[21]  J. Peelle.  Listening Effort: How the Cognitive Consequences of Acoustic Challenge Are Reflected in Brain and Behavior , 2017, Ear and hearing.

[22]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[23]  J. Schoffelen,et al.  Beta oscillations relate to the N400m during language comprehension , 2012, Human brain mapping.

[24]  Peter Hagoort,et al.  MUC (Memory, Unification, Control) and beyond , 2013, Front. Psychol..

[25]  J Gross,et al.  REPRINTS , 1962, The Lancet.

[26]  R. Krauss,et al.  Do conversational hand gestures communicate? , 1991, Journal of personality and social psychology.

[27]  Riitta Hari,et al.  Actor's and observer's primary motor cortices stabilize similarly after seen or heard motor actions , 2007, Proceedings of the National Academy of Sciences.

[28]  Asli Ozyurek.  Hearing and seeing meaning in speech and gesture: insights from brain and behaviour , 2014 .

[29]  A. Friederici,et al.  On the lateralization of emotional prosody: An event-related functional MR investigation , 2003, Brain and Language.

[30]  Christoph M. Michel,et al.  Left temporal alpha-band activity reflects single word intelligibility , 2013, Front. Syst. Neurosci..

[31]  Marcel Bastiaansen,et al.  Integration or Predictability? A Further Specification of the Functional Role of Gamma Oscillations in Language Comprehension , 2012, Front. Psychology.

[32]  Zeshu Shao,et al.  The Role of Synchrony and Ambiguity in Speech–Gesture Integration during Comprehension , 2011, Journal of Cognitive Neuroscience.

[33]  Robert Oostenveld,et al.  Enhanced EEG gamma-band activity reflects multisensory semantic matching in visual-to-auditory object priming , 2008, NeuroImage.

[34]  Mario Dzemidzic,et al.  Hemispheric roles in the perception of speech prosody , 2004, NeuroImage.

[35]  Jonas Obleser,et al.  Alpha Rhythms in Audition: Cognitive and Clinical Perspectives , 2011, Front. Psychology.

[36]  Robert Oostenveld,et al.  Online and offline tools for head movement compensation in MEG , 2013, NeuroImage.

[37]  Andreas K. Engel,et al.  Noise alters beta-band activity in superior temporal cortex during audiovisual speech processing , 2013, NeuroImage.

[38]  Robert Oostenveld,et al.  Motor-cortical beta oscillations are modulated by correctness of observed action , 2008, NeuroImage.

[39]  F. L. D. Silva,et al.  Event-related EEG/MEG synchronization and desynchronization: basic principles , 1999, Clinical Neurophysiology.

[40]  Adam R. Walczak,et al.  At the heart of the ventral attention system: The right anterior insula , 2009, Human brain mapping.

[41]  Alfons Schnitzler,et al.  Language–motor interference reflected in MEG beta oscillations , 2015, NeuroImage.

[42]  Thomas C. Gunter,et al.  The benefit of gestures during communication: Evidence from hearing and hearing-impaired individuals , 2012, Cortex.

[43]  W. Singer,et al.  The gamma cycle , 2007, Trends in Neurosciences.

[44]  O. Bertrand,et al.  Oscillatory gamma activity in humans and its role in object representation , 1999, Trends in Cognitive Sciences.

[45]  Anjali Raja Beharelle,et al.  Frontal and temporal contributions to understanding the iconic co‐speech gestures that accompany speech , 2014, Human brain mapping.

[46]  Jonas Obleser,et al.  Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions , 2010, NeuroImage.

[47]  P. Hagoort,et al.  Language beyond action , 2008, Journal of Physiology-Paris.

[48]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[49]  P. Mitra,et al.  Analysis of dynamic brain imaging data. , 1998, Biophysical journal.

[50]  R. Zatorre,et al.  Structure and function of auditory cortex: music and speech , 2002, Trends in Cognitive Sciences.

[51]  J. Vrba,et al.  Signal processing in magnetoencephalography. , 2001, Methods.

[52]  Jonas Obleser,et al.  The Brain Dynamics of Rapid Perceptual Adaptation to Adverse Listening Conditions , 2013, The Journal of Neuroscience.

[53]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[54]  S. Scott,et al.  Identification of a pathway for intelligible speech in the left temporal lobe. , 2000, Brain : a journal of neurology.

[55]  Burkhard Maess,et al.  Adverse Listening Conditions and Memory Load Drive a Common Alpha Oscillatory Network , 2012, The Journal of Neuroscience.

[56]  A. Engel,et al.  Cognitive functions of gamma-band activity: memory match and utilization , 2004, Trends in Cognitive Sciences.

[57]  Heather Shovelton,et al.  Mapping the Range of Information Contained in the Iconic Hand Gestures that Accompany Spontaneous Speech , 1999 .

[58]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[59]  M. Ernestus,et al.  Alpha and gamma band oscillations index differential processing of acoustically reduced and full forms , 2016, Brain and Language.

[60]  R V Shannon,et al.  Speech Recognition with Primarily Temporal Cues , 1995, Science.

[61]  Stefanie E. Kuchinsky,et al.  The Cingulo-Opercular Network Provides Word-Recognition Benefit , 2013, The Journal of Neuroscience.

[62]  Sabine Weiss,et al.  “Too Many betas do not Spoil the Broth”: The Role of Beta Brain Oscillations in Language Processing , 2012, Front. Psychology.

[63]  A. Engel,et al.  Spectral fingerprints of large-scale neuronal interactions , 2012, Nature Reviews Neuroscience.

[64]  Andreas K. Engel,et al.  Oscillatory Synchronization in Large-Scale Cortical Networks Predicts Perception , 2011, Neuron.

[65]  O. Jensen,et al.  Shaping Functional Architecture by Oscillatory Alpha Activity: Gating by Inhibition , 2010, Front. Hum. Neurosci..

[66]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[67]  D. Perrett,et al.  A region of right posterior superior temporal sulcus responds to observed intentional actions , 2004, Neuropsychologia.

[68]  Aslı Özyürek,et al.  Hearing and seeing meaning in speech and gesture: insights from brain and behaviour , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.