Decoding the Real-Time Neurobiological Properties of Incremental Semantic Interpretation

Abstract Communication through spoken language is a central human capacity, involving a wide range of complex computations that incrementally interpret each word into meaningful sentences. However, surprisingly little is known about the spatiotemporal properties of the complex neurobiological systems that support these dynamic predictive and integrative computations. Here, we focus on prediction, a core incremental processing operation guiding the interpretation of each upcoming word with respect to its preceding context. To investigate the neurobiological basis of how semantic constraints change and evolve as each word in a sentence accumulates over time, in a spoken sentence comprehension study, we analyzed the multivariate patterns of neural activity recorded by source-localized electro/magnetoencephalography (EMEG), using computational models capturing semantic constraints derived from the prior context on each upcoming word. Our results provide insights into predictive operations subserved by different regions within a bi-hemispheric system, which over time generate, refine, and evaluate constraints on each word as it is heard.

[1]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[2]  Katherine A. DeLong,et al.  Probabilistic word pre-activation during language comprehension inferred from electrical brain activity , 2005, Nature Neuroscience.

[3]  John Hale,et al.  Uncertainty About the Rest of the Sentence , 2006, Cogn. Sci..

[4]  Kara D. Federmeier,et al.  Right words and left words: electrophysiological evidence for hemispheric differences in meaning processing. , 1999, Brain research. Cognitive brain research.

[5]  S. Cappa,et al.  The neural substrate of naming events: effects of processing demands but not of grammatical class. , 2008, Cerebral cortex.

[6]  Ellen F. Lau,et al.  A cortical network for semantics: (de)constructing the N400 , 2008, Nature Reviews Neuroscience.

[7]  P. McGuire,et al.  Engagement of right temporal cortex during processing of linguistic context , 2001, Neuropsychologia.

[8]  W. Marslen-Wilson Functional parallelism in spoken word-recognition , 1987, Cognition.

[9]  Gregory Hickok,et al.  The Cortical Organization of Syntax. , 2019, Cerebral Cortex.

[10]  S. Taulu,et al.  Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements , 2006, Physics in medicine and biology.

[11]  G H Glover,et al.  Functional magnetic resonance imaging evidence for right-hemisphere involvement in processing unusual semantic relationships. , 2000, Neuropsychology.

[12]  Ellen F. Lau,et al.  fMRI evidence that left posterior temporal cortex contributes to N400 effects of predictability independent of congruity , 2019, Brain and Language.

[13]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[14]  G. Altmann,et al.  The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements , 2003 .

[15]  L. Tyler,et al.  Decoding the Cortical Dynamics of Sound-Meaning Mapping , 2017, The Journal of Neuroscience.

[16]  W. Marslen-Wilson,et al.  Integrative Processes in Utterance Resolution , 1993 .

[17]  R. H. Baayen,et al.  The CELEX Lexical Database (CD-ROM) , 1996 .

[18]  R. Oostenveld,et al.  Nonparametric statistical testing of EEG- and MEG-data , 2007, Journal of Neuroscience Methods.

[19]  Edward M. Bowden,et al.  Right and Left Hemisphere Cooperation for Drawing Predictive and Coherence Inferences during Normal Story Comprehension , 2000, Brain and Language.

[20]  Lorraine K Tyler,et al.  Morphology, language and the brain: the decompositional substrate for language comprehension , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[21]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[22]  Steven G. Luke,et al.  Limits on lexical prediction during reading , 2016, Cognitive Psychology.

[23]  W D Marslen-Wilson,et al.  Sentence Perception as an Interactive Parallel Process , 1975, Science.

[24]  Gina R Kuperberg,et al.  Separate streams or probabilistic inference? What the N400 can tell us about the comprehension of events , 2016, Language, cognition and neuroscience.

[25]  Peter,et al.  Semantic Unification , 2008 .

[26]  Marcel Adam Just,et al.  From the SelectedWorks of Marcel Adam Just 2007 Lexical ambiguity in sentence comprehension , 2016 .

[27]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[28]  Sharon L. Thompson-Schill,et al.  Dissociating semantic and phonological maintenance using fMRI , 2004, Cognitive, affective & behavioral neuroscience.

[29]  C. Price The anatomy of language: a review of 100 fMRI studies published in 2009 , 2010, Annals of the New York Academy of Sciences.

[30]  Antal van den Bosch,et al.  Prediction During Natural Language Comprehension. , 2016, Cerebral cortex.

[31]  Roberto G. de Almeida,et al.  The Neuronal Correlates of Indeterminate Sentence Comprehension: An fMRI Study , 2016, Front. Hum. Neurosci..

[32]  William D. Marslen-Wilson,et al.  The On-Line Effects of Semantic Context on Syntactic Processing , 1977 .

[33]  L. Tyler,et al.  Differentiating Hemispheric Contributions to Syntax and Semantics in Patients with Left-Hemisphere Lesions , 2012, The Journal of Neuroscience.

[34]  M. Jung-Beeman Bilateral brain processes for comprehending natural language , 2005, Trends in Cognitive Sciences.

[35]  Peter Hagoort,et al.  MUC (Memory, Unification, Control) and beyond , 2013, Front. Psychol..

[36]  M. Schlesewsky,et al.  Reconciling time, space and function: A new dorsal–ventral stream model of sentence comprehension , 2013, Brain and Language.

[37]  Derek K. Jones,et al.  Perisylvian language networks of the human brain , 2005, Annals of neurology.

[38]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[39]  W. Marslen-Wilson,et al.  The temporal structure of spoken language understanding , 1980, Cognition.

[40]  A. Friederici,et al.  Differential cortical contribution of syntax and semantics: An fMRI study on two-word phrasal processing , 2017, Cortex.

[41]  Kara D. Federmeier,et al.  Finding the right word: Hemispheric asymmetries in the use of sentence context information , 2007, Neuropsychologia.

[42]  M. Beeman,et al.  Complementary Right- and Left-Hemisphere Language Comprehension , 1998 .

[43]  Colin Humphries,et al.  Time course of semantic processes during sentence comprehension: An fMRI study , 2007, NeuroImage.

[44]  Gerry T M Altmann,et al.  Abstraction and generalization in statistical learning: implications for the relationship between semantic types and episodic tokens , 2017, Philosophical Transactions of the Royal Society B: Biological Sciences.

[45]  Karl J. Friston,et al.  Algorithmic procedures for Bayesian MEG/EEG source reconstruction in SPM , 2014, NeuroImage.

[46]  Ted Briscoe,et al.  A Large Subcategorization Lexicon for Natural Language Processing Applications , 2006, LREC.

[47]  Roger Levy,et al.  Cloze but no cigar: The complex relationship between cloze, corpus, and subjective probabilities in language processing , 2011, CogSci.

[48]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[49]  Alex Clarke,et al.  Neural dynamics of semantic composition , 2019, Proceedings of the National Academy of Sciences.

[50]  John Hale,et al.  A Probabilistic Earley Parser as a Psycholinguistic Model , 2001, NAACL.

[51]  Jeffrey L Elman,et al.  Lexical knowledge without a lexicon? , 2011, The mental lexicon.

[52]  Sylvain Baillet,et al.  Two Distinct Neural Timescales for Predictive Speech Processing , 2019, Neuron.

[53]  Lorraine K Tyler,et al.  Language-related domain-specific and domain-general systems in the human brain , 2018, Current Opinion in Behavioral Sciences.

[54]  Alessandro Lenci,et al.  Distributional Memory: A General Framework for Corpus-Based Semantics , 2010, CL.

[55]  S. Cappa,et al.  Nouns and verbs in the brain: A review of behavioural, electrophysiological, neuropsychological and imaging studies , 2011, Neuroscience & Biobehavioral Reviews.

[56]  Li Su,et al.  Spatiotemporal Searchlight Representational Similarity Analysis in EMEG Source Space , 2012, 2012 Second International Workshop on Pattern Recognition in NeuroImaging.

[57]  Gina R Kuperberg,et al.  What do we mean by prediction in language comprehension? , 2016, Language, cognition and neuroscience.

[58]  Billi Randall,et al.  Balancing Prediction and Sensory Input in Speech Comprehension: The Spatiotemporal Dynamics of Word Recognition in Context , 2018, The Journal of Neuroscience.

[59]  Marta Kutas,et al.  Pre-Processing in Sentence Comprehension: Sensitivity to Likely Upcoming Meaning and Structure , 2014, Lang. Linguistics Compass.

[60]  Cathy J. Price,et al.  A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading , 2012, NeuroImage.

[61]  R. Levy Expectation-based syntactic comprehension , 2008, Cognition.

[62]  Matthew H. Davis,et al.  The neural mechanisms of speech comprehension: fMRI studies of semantic ambiguity. , 2005, Cerebral cortex.

[63]  J. Elman,et al.  Effects of event knowledge in processing verbal arguments. , 2010, Journal of memory and language.

[64]  Kara D. Federmeier,et al.  What's 'Right' in Language Comprehension: Event-Related Potentials Reveal Right Hemisphere Language Capabilities , 2008, Lang. Linguistics Compass.

[65]  A. Friederici The brain basis of language processing: from structure to function. , 2011, Physiological reviews.

[66]  S. Bookheimer Functional MRI of language: new approaches to understanding the cortical organization of semantic processing. , 2002, Annual review of neuroscience.

[67]  Karl J. Friston,et al.  Canonical Source Reconstruction for MEG , 2007, Comput. Intell. Neurosci..

[68]  Kara D. Federmeier,et al.  Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). , 2011, Annual review of psychology.

[69]  Jelena Mirkovic,et al.  Incrementality and Prediction in Human Sentence Processing , 2009, Cogn. Sci..

[70]  M. Kutas,et al.  Semantic integration in reading: engagement of the right hemisphere during discourse processing. , 1999, Brain : a journal of neurology.

[71]  Diego L. Lorca-Puls,et al.  How right hemisphere damage after stroke can impair speech comprehension , 2018, Brain : a journal of neurology.

[72]  Mark Steyvers,et al.  Finding scientific topics , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[73]  Sarah Brown-Schmidt,et al.  Hippocampal Contributions to Language Use and Processing , 2017 .