Frequency-specific brain dynamics related to prediction during language comprehension

The brain's remarkable capacity to process spoken language virtually in real time requires fast and efficient information processing machinery. In this study, we investigated how frequency-specific brain dynamics relate to models of probabilistic language prediction during auditory narrative comprehension. We recorded MEG activity while participants were listening to auditory stories in Dutch. Using trigram statistical language models, we estimated for every word in a story its conditional probability of occurrence. On the basis of word probabilities, we computed how unexpected the current word is given its context (word perplexity) and how (un)predictable the current linguistic context is (word entropy). We then evaluated whether source-reconstructed MEG oscillations at different frequency bands are modulated as a function of these language processing metrics. We show that theta-band source dynamics are increased in high relative to low entropy states, likely reflecting lexical computations. Beta-band dynamics are increased in situations of low word entropy and perplexity possibly reflecting maintenance of ongoing cognitive context. These findings lend support to the idea that the brain engages in the active generation and evaluation of predicted language based on the statistical properties of the input signal.

[1]  Karl J. Friston,et al.  Canonical Microcircuits for Predictive Coding , 2012, Neuron.

[2]  R. Dolan,et al.  Knowing how much you don't know: a neural organization of uncertainty estimates , 2012, Nature Reviews Neuroscience.

[3]  Colin Klein,et al.  Ghosts in machine learning for cognitive neuroscience: Moving from data to theory , 2017, NeuroImage.

[4]  Lars Meyer,et al.  The neural oscillations of speech processing and language comprehension: state of the art and emerging mechanisms , 2018, The European journal of neuroscience.

[5]  Gina R Kuperberg,et al.  What do we mean by prediction in language comprehension? , 2016, Language, cognition and neuroscience.

[6]  Peter Hagoort,et al.  Frequency-based Segregation of Syntactic and Semantic Unification during Online Sentence Level Language Comprehension , 2015, Journal of Cognitive Neuroscience.

[7]  Ton Dijkstra,et al.  Theta Responses Are Involved in LexicalSemantic Retrieval during Language Processing , 2005, Journal of Cognitive Neuroscience.

[8]  A. Pouget,et al.  Probabilistic brains: knowns and unknowns , 2013, Nature Neuroscience.

[9]  Floris P. de Lange,et al.  How Prediction Errors Shape Perception, Attention, and Motivation , 2012, Front. Psychology.

[10]  Stanislas Dehaene,et al.  Neurophysiological dynamics of phrase-structure building during sentence processing , 2017, Proceedings of the National Academy of Sciences.

[11]  D. Poeppel,et al.  Phase Patterns of Neuronal Responses Reliably Discriminate Speech in Human Auditory Cortex , 2007, Neuron.

[12]  Antal van den Bosch,et al.  Memory-Based Machine Translation and Language Modeling , 2009, Prague Bull. Math. Linguistics.

[13]  Christopher D. Manning,et al.  Probabilistic models of language processing and acquisition , 2006, Trends in Cognitive Sciences.

[14]  Sabine Weiss,et al.  “Too Many betas do not Spoil the Broth”: The Role of Beta Brain Oscillations in Language Processing , 2012, Front. Psychology.

[15]  Joachim Gross,et al.  Auditory cortical delta-entrainment interacts with oscillatory power in multiple fronto-parietal networks , 2017, NeuroImage.

[16]  Nelleke Oostdijk,et al.  The Spoken Dutch Corpus. Overview and First Evaluation , 2000, LREC.

[17]  W. Marslen-Wilson Functional parallelism in spoken word-recognition , 1987, Cognition.

[18]  Stefano Panzeri,et al.  Contributions of local speech encoding and functional connectivity to audio-visual speech perception , 2017, eLife.

[19]  Peter Hagoort,et al.  The neurobiology of language beyond single words. , 2014, Annual review of neuroscience.

[20]  Antal van den Bosch,et al.  Using stochastic language models (SLM) to map lexical, syntactic, and phonological information processing in the brain , 2017, PloS one.

[21]  Jean-Pierre Martens,et al.  Word Segmentation in the Spoken Dutch Corpus , 2002, LREC.

[22]  G. Ermentrout,et al.  Gamma rhythms and beta rhythms have different synchronization properties. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Frank Keller,et al.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity , 2008, Cognition.

[24]  Y. Luo,et al.  Electroencephalogram oscillations differentiate semantic and prosodic processes during sentence reading , 2010, Neuroscience.

[25]  A. Bachrach Imaging neural correlates of syntactic complexity in a naturalistic context , 2008 .

[26]  Marcel Bastiaansen,et al.  Integration or Predictability? A Further Specification of the Functional Role of Gamma Oscillations in Language Comprehension , 2012, Front. Psychology.

[27]  Roel M. Willems,et al.  Word predictability and semantic similarity show distinct patterns of brain activity during language comprehension , 2017 .

[28]  Robin A. A. Ince,et al.  Frontal Top-Down Signals Increase Coupling of Auditory Low-Frequency Oscillations to Continuous Speech in Human Listeners , 2015, Current Biology.

[29]  Andreas Stolcke,et al.  SRILM - an extensible language modeling toolkit , 2002, INTERSPEECH.

[30]  E Emmanu SUBTLEX-NL: A new measure for Dutch word frequency based on film subtitles , 2010 .

[31]  Edmund C. Lalor,et al.  The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous Stimuli , 2016, Front. Hum. Neurosci..

[32]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[33]  Evelyn C. Ferstl,et al.  The extended language network: A meta‐analysis of neuroimaging studies on text comprehension , 2008, Human brain mapping.

[34]  Roel M. Willems,et al.  On the other hand: including left-handers in cognitive neuroscience and neurogenetics , 2014, Nature Reviews Neuroscience.

[35]  Antal van den Bosch,et al.  Prediction During Natural Language Comprehension. , 2016, Cerebral cortex.

[36]  Peter Hagoort,et al.  Neural activity during sentence processing as reflected in theta, alpha, beta, and gamma oscillations , 2016, NeuroImage.

[37]  Nivedita Mani,et al.  Is prediction necessary to understand language? Probably not , 2016 .

[38]  John Hale,et al.  Entropy Reduction correlates with temporal lobe activity , 2017, CMCL@EACL.

[39]  Karl J. Friston,et al.  A theory of cortical responses , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[40]  P. Hagoort,et al.  Oscillatory neuronal dynamics during language comprehension. , 2006, Progress in brain research.

[41]  Mark W. Woolrich,et al.  FSL , 2012, NeuroImage.

[42]  César F. Lima,et al.  Roles of Supplementary Motor Areas in Auditory Processing and Auditory Imagery , 2016, Trends in Neurosciences.

[43]  Ninon Burgos,et al.  New advances in the Clinica software platform for clinical neuroimaging studies , 2019 .

[44]  Joseph T. Devlin,et al.  The hearing ear is always found close to the speaking tongue: Review of the role of the motor system in speech perception , 2017, Brain and Language.

[45]  Peter Hagoort,et al.  Syntactic Processing Modulates the θ Rhythm of the Human EEG , 2002, NeuroImage.

[46]  Peter Hagoort,et al.  Robust neuronal oscillatory entrainment to speech displays individual variation in lateralisation , 2018 .

[47]  Roel M. Willems,et al.  Neuroscience and Biobehavioral Reviews , 2022 .

[48]  Robert Oostenveld,et al.  Online and offline tools for head movement compensation in MEG , 2013, NeuroImage.

[49]  Adrian Staub,et al.  The Effect of Lexical Predictability on Eye Movements in Reading: Critical Review and Theoretical Interpretation , 2015, Lang. Linguistics Compass.

[50]  P. Hagoort,et al.  EEG theta and gamma responses to semantic violations in online sentence processing , 2006, Brain and Language.

[51]  Mante S. Nieuwland,et al.  Large-scale replication study reveals a limit on probabilistic prediction in language comprehension , 2018, eLife.

[52]  Peter Hagoort,et al.  Syntactic Unification Operations Are Reflected in Oscillatory Dynamics during On-line Sentence Comprehension , 2010, Journal of Cognitive Neuroscience.

[53]  Uri Hasson,et al.  The neurobiology of uncertainty: implications for statistical learning , 2017, Philosophical Transactions of the Royal Society B: Biological Sciences.

[54]  James L. McClelland,et al.  Modelling the N400 brain potential as change in a probabilistic representation of meaning , 2018, Nature Human Behaviour.

[55]  P. Schyns,et al.  Speech Rhythms and Multiplexed Oscillatory Sensory Coding in the Human Brain , 2013, PLoS biology.

[56]  John Hale,et al.  Finding syntax in human encephalography with beam search , 2018, ACL.

[57]  W. Drongelen,et al.  Localization of brain electrical activity via linearly constrained minimum variance spatial filtering , 1997, IEEE Transactions on Biomedical Engineering.

[58]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[59]  John M. Henderson,et al.  Language structure in the brain: A fixation-related fMRI study of syntactic surprisal in reading , 2016, NeuroImage.

[60]  Herbert Schriefers,et al.  Discourse-level semantic coherence influences beta oscillatory dynamics and the N400 during sentence comprehension , 2017 .

[61]  Stephen M Smith,et al.  Fast robust automated brain extraction , 2002, Human brain mapping.

[62]  J Gross,et al.  REPRINTS , 1962, The Lancet.

[63]  R. Shillcock,et al.  Eye Movements Reveal the On-Line Computation of Lexical Probabilities During Reading , 2003, Psychological science.

[64]  G. Nolte The magnetic lead field theorem in the quasi-static approximation and its use for magnetoencephalography forward calculation in realistic volume conductors. , 2003, Physics in medicine and biology.

[65]  Tal Linzen,et al.  The role of morphology in phoneme prediction: Evidence from MEG , 2014, Brain and Language.

[66]  James H. Martin,et al.  Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition , 2000 .

[67]  Sylvain Baillet,et al.  Motor origin of temporal predictions in auditory attention , 2017, Proceedings of the National Academy of Sciences.

[68]  Milena Rabovsky,et al.  Simulating the N400 ERP component as semantic network error: Insights from a feature-based connectionist attractor model of word meaning , 2014, Cognition.

[69]  Zachary M. Smith,et al.  Chimaeric sounds reveal dichotomies in auditory perception , 2002, Nature.

[70]  E Ahissar,et al.  Speech comprehension is correlated with temporal response patterns recorded from auditory cortex , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[71]  S. Frank,et al.  Insensitivity of the Human Sentence-Processing System to Hierarchical Structure , 2011, Psychological science.

[72]  J. Schoffelen,et al.  Nonparametric statistical testing of coherence differences , 2007, Journal of Neuroscience Methods.

[73]  Roger Levy,et al.  Cloze but no cigar: The complex relationship between cloze, corpus, and subjective probabilities in language processing , 2011, CogSci.

[74]  Kara D. Federmeier,et al.  Alpha and theta band dynamics related to sentential constraint and word expectancy , 2017, Language, cognition and neuroscience.

[75]  Nathaniel J. Smith,et al.  The effect of word predictability on reading time is logarithmic , 2013, Cognition.

[76]  A. Engel,et al.  Beta-band oscillations—signalling the status quo? , 2010, Current Opinion in Neurobiology.

[77]  Jed A. Meltzer,et al.  Localization of electrophysiological responses to semantic and syntactic anomalies in language comprehension with MEG , 2015, NeuroImage.

[78]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[79]  John Hale,et al.  Information-theoretical Complexity Metrics , 2016, Lang. Linguistics Compass.

[80]  Jonathan Brennan,et al.  Naturalistic Sentence Comprehension in the Brain , 2016, Lang. Linguistics Compass.

[81]  Roel M. Willems,et al.  Early decreases in alpha and gamma band power distinguish linguistic from visual information during spoken sentence comprehension , 2008, Brain Research.

[82]  J. Tenenbaum,et al.  Special issue on “Probabilistic models of cognition , 2022 .

[83]  Walter Daelemans,et al.  An efficient memory-based morphosyntactic tagger and parser for Dutch , 2007, CLIN 2007.

[84]  Saskia Haegens,et al.  Beyond the Status Quo: A Role for Beta Oscillations in Endogenous Content (Re)Activation , 2017, eNeuro.

[85]  Moshe Bar,et al.  Predictions: a universal principle in the operation of the human brain , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[86]  W. Luh,et al.  Abstract linguistic structure correlates with temporal activity during naturalistic comprehension , 2016, Brain and Language.

[87]  Raymond J. Dolan,et al.  Information theory, novelty and hippocampal responses: unpredicted or unpredictable? , 2005, Neural Networks.

[88]  W. Bechtel,et al.  Marr’s Computational Level and Delineating Phenomena , 2018 .

[89]  M. Bastiaansen,et al.  A predictive coding framework for rapid neural dynamics during sentence-level language comprehension , 2015, Cortex.

[90]  J. Schoffelen,et al.  Beta oscillations relate to the N400m during language comprehension , 2012, Human brain mapping.

[91]  A. Clark Whatever next? Predictive brains, situated agents, and the future of cognitive science. , 2013, The Behavioral and brain sciences.

[92]  D. Knill,et al.  The Bayesian brain: the role of uncertainty in neural coding and computation , 2004, Trends in Neurosciences.

[93]  Roland Schäfer,et al.  Building Large Corpora from the Web Using a New Efficient Tool Chain , 2012, LREC.

[94]  Robert Oostenveld,et al.  FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data , 2010, Comput. Intell. Neurosci..

[95]  C. Ranganath,et al.  Two cortical systems for memory-guided behaviour , 2012, Nature Reviews Neuroscience.

[96]  Marcus Kaiser,et al.  Temporal Interactions between Cortical Rhythms , 2008, Front. Neurosci..

[97]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.