Robust effects of working memory demand during naturalistic language comprehension in language-selective cortex

Abstract A standard view of human language processing is that comprehenders build richly structured mental representations of natural language utterances, word by word, using computationally costly memory operations supported by domain-general working memory resources. However, three core claims of this view have been questioned, with some prior work arguing that (1) rich word-by-word structure building is not a core function of the language comprehension system, (2) apparent working memory costs are underlyingly driven by word predictability (surprisal), and/or (3) language comprehension relies primarily on domain-general rather than domain-specific working memory resources. In this work, we simultaneously evaluate all three of these claims using naturalistic comprehension in fMRI. In each participant, we functionally localize (a) a language-selective network and (b) a ‘multiple-demand’ network that supports working memory across domains, and we analyze the responses in these two networks of interest during naturalistic story listening with respect to a range of theory-driven predictors of working memory demand under rigorous surprisal controls. Results show robust surprisal-independent effects of word-by-word memory demand in the language network and no effect of working memory demand in the multiple demand network. Our findings thus support the view that language comprehension (1) entails word-by-word structure building using (2) computationally intensive memory operations that are not explained by surprisal. However, these results challenge (3) the domain-generality of the resources that support these operations, instead indicating that working memory operations for language comprehension are carried out by the same neural resources that store linguistic knowledge. Significance Statement This study uses fMRI to investigate signatures of working memory (WM) demand during naturalistic story listening, using a broad range of theoretically motivated estimates of WM demand. Results support a strong effect of WM demand in language-selective brain regions but no effect of WM demand in “multiple demand” regions that have previously been associated with WM in non-linguistic domains. We further show evidence that WM effects in language regions are distinct from effects of word predictability. Our findings support a core role for WM in incremental language processing, using WM resources that are specialized for language.

[1]  William Schuler,et al.  Memory access during incremental sentence processing causes reading time latency , 2016, CL4LC@COLING 2016.

[2]  Larry R Squire,et al.  Intact Working Memory for Relational Information after Medial Temporal Lobe Damage , 2010, The Journal of Neuroscience.

[3]  Nancy Kanwisher,et al.  Broad domain generality in focal regions of frontal and parietal cortex , 2013, Proceedings of the National Academy of Sciences.

[4]  Evelina Fedorenko,et al.  A new fun and robust version of an fMRI localizer for the frontotemporal language system , 2017, Cognitive neuroscience.

[5]  Lorraine K Tyler,et al.  Language-related domain-specific and domain-general systems in the human brain , 2018, Current Opinion in Behavioral Sciences.

[6]  P. Johnson-Laird,et al.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness , 1985 .

[7]  P. Gordon,et al.  Similarity-based interference during language comprehension: Evidence from eye tracking during reading. , 2006, Journal of experimental psychology. Learning, memory, and cognition.

[8]  J. D. E. Gabrieli,et al.  Integration of diverse information in working memory within the frontal lobe , 2000, Nature Neuroscience.

[9]  M. Leszczyński.  How Does Hippocampus Contribute to Working Memory Processing? , 2011, Front. Hum. Neurosci..

[10]  Antal van den Bosch,et al.  Using stochastic language models (SLM) to map lexical, syntactic, and phonological information processing in the brain , 2017, PloS one.

[11]  John T Hale,et al.  Hierarchical structure guides rapid linguistic predictions during naturalistic listening , 2019, PloS one.

[12]  Barbara Landau,et al.  The Necessity of the Medial Temporal Lobe for Statistical Learning , 2014, Journal of Cognitive Neuroscience.

[13]  Richard Futrell,et al.  Lossy‐Context Surprisal: An Information‐Theoretic Model of Memory Effects in Sentence Processing , 2020, Cogn. Sci..

[14]  Y. Miyashita,et al.  Image, language, brain , 2000 .

[15]  Roger Levy,et al.  Sequential vs. Hierarchical Syntactic Models of Human Incremental Sentence Processing , 2012, CMCL@NAACL-HLT.

[16]  Ingrid R. Olson,et al.  Working Memory for Conjunctions Relies on the Medial Temporal Lobe , 2006, The Journal of Neuroscience.

[17]  Matthew H. Davis,et al.  Is the link between anatomical structure and function equally strong at all cognitive levels of processing? , 2012, Cerebral cortex.

[18]  Bruce L. Miller,et al.  Anatomical Correlates of Sentence Comprehension and Verbal Working Memory in Neurodegenerative Disease , 2007, The Journal of Neuroscience.

[19]  S. Gershman,et al.  Memory as a Computational Resource , 2021, Trends in Cognitive Sciences.

[20]  Adrian T. Lee,et al.  Discrimination of Large Venous Vessels in Time‐Course Spiral Blood‐Oxygen‐Level‐Dependent Magnetic‐Resonance Functional Neuroimaging , 1995, Magnetic resonance in medicine.

[21]  Angela D. Friederici,et al.  Left parietal alpha enhancement during working memory-intensive sentence processing , 2013, Cortex.

[22]  Gregory Hickok,et al.  The Cortical Organization of Syntax. , 2019, Cerebral cortex.

[23]  Evelina Fedorenko,et al.  Lack of selectivity for syntax relative to word meanings throughout the language network , 2020, Cognition.

[24]  J. Duncan,et al.  Integrated Intelligence from Distributed Brain Activity , 2020, Trends in Cognitive Sciences.

[25]  Cory Shain,et al.  A large-scale study of the effects of word frequency and predictability in naturalistic reading , 2019, NAACL.

[26]  N. Kanwisher,et al.  New method for fMRI investigations of language: defining ROIs functionally in individual subjects. , 2010, Journal of neurophysiology.

[27]  Walter Schneider,et al.  The cognitive control network: Integrated cortical regions with dissociable functions , 2007, NeuroImage.

[28]  Jonathan Brennan,et al.  Naturalistic Sentence Comprehension in the Brain , 2016, Lang. Linguistics Compass.

[29]  R. Binnick.  Time and the Verb: A Guide to Tense and Aspect , 1991 .

[30]  Tal Linzen,et al.  A Neural Model of Adaptation in Reading , 2018, EMNLP.

[31]  M. Just,et al.  From the SelectedWorks of Marcel Adam Just 1992 A capacity theory of comprehension : Individual differences in working memory , 2017 .

[32]  P. Gordon,et al.  Memory interference during language processing. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[33]  Edward Gibson,et al.  Online Syntactic Storage Costs in Sentence Comprehension. , 2005 .

[34]  Christopher Jarrold,et al.  Working Memory and the Hippocampus , 2011, Journal of Cognitive Neuroscience.

[35]  Evelina Fedorenko,et al.  A robust dissociation among the language, multiple demand, and default mode networks: Evidence from inter-region correlations in effect size , 2017, Neuropsychologia.

[36]  Daniel J. Rosenkrantz,et al.  Deterministic Left Corner Parsing (Extended Abstract) , 1970, SWAT.

[37]  Philipp Koehn,et al.  Scalable Modified Kneser-Ney Language Model Estimation , 2013, ACL.

[38]  C. Honey,et al.  Hierarchical process memory: memory as an integral component of information processing , 2015, Trends in Cognitive Sciences.

[39]  Lauren M DiNicola,et al.  Situating the Left-Lateralized Language Network in the Broader Organization of Multiple Specialized Large-Scale Distributed Networks. , 2020, Journal of neurophysiology.

[40]  Ramona O Hopkins,et al.  Working Memory and the Organization of Brain Systems , 2008, The Journal of Neuroscience.

[41]  William Schuler,et al.  A Model of Language Processing as Hierarchic Sequential Prediction , 2013, Top. Cogn. Sci..

[42]  Antal van den Bosch,et al.  Frequency-specific brain dynamics related to prediction during language comprehension , 2018, NeuroImage.

[43]  Faraneh Vargha-Khadem,et al.  Is the hippocampus necessary for visual and verbal binding in working memory? , 2010, Neuropsychologia.

[44]  John Hale,et al.  Localising memory retrieval and syntactic composition: an fMRI study of naturalistic language comprehension , 2018, Language, Cognition and Neuroscience.

[45]  D Rudrauf,et al.  Distributed neural system for general intelligence revealed by lesion mapping , 2010, Proceedings of the National Academy of Sciences.

[46]  Christopher J. Honey,et al.  Future trends in Neuroimaging: Neural processes as expressed within real-life contexts , 2012, NeuroImage.

[47]  Stefan L. Frank,et al.  Evaluating information-theoretic measures of word prediction in naturalistic sentence reading , 2019, Neuropsychologia.

[48]  Ellen F. Lau,et al.  The role of the IFG and pSTS in syntactic prediction: Evidence from a parametric study of hierarchical structure in fMRI , 2017, Cortex.

[49]  William Schuler,et al.  fMRI reveals language-specific predictive coding during naturalistic sentence comprehension , 2020, Neuropsychologia.

[50]  N Makris,et al.  Location of lesions in stroke patients with deficits in syntactic processing in sentence comprehension. , 1996, Brain : a journal of neurology.

[51]  E. Gibson.  The dependency locality theory: A distance-based theory of linguistic complexity. , 2000 .

[52]  Alfred Anwander,et al.  Segregating the core computational faculty of human language from working memory , 2009, Proceedings of the National Academy of Sciences.

[53]  W. Luh,et al.  Abstract linguistic structure correlates with temporal activity during naturalistic comprehension , 2016, Brain and Language.

[54]  Laura E. Suárez,et al.  Gradients of structure–function tethering across neocortex , 2019, Proceedings of the National Academy of Sciences.

[55]  Yosef Grodzinsky,et al.  Working memory and syntax interact in Broca's area , 2007, NeuroImage.

[56]  J. Duncan,et al.  Common regions of the human frontal lobe recruited by diverse cognitive demands , 2000, Trends in Neurosciences.

[57]  John Duncan,et al.  A Domain-General Cognitive Core Defined in Multimodally Parcellated Human Cortex , 2020, Cerebral cortex.

[58]  A. Staub,et al.  Maintenance cost in the processing of subject-verb dependencies. , 2021, Journal of experimental psychology. Learning, memory, and cognition.

[59]  C. Clifton,et al.  Underspecification of syntactic ambiguities: Evidence from self-paced reading , 2008, Memory & cognition.

[60]  G. Waters,et al.  Verbal working memory and sentence comprehension , 1999, Behavioral and Brain Sciences.

[61]  S. Thompson-Schill,et al.  Reworking the language network , 2014, Trends in Cognitive Sciences.

[62]  R. Malach,et al.  Syntactic structure building in the anterior temporal lobe during natural story listening , 2012, Brain and Language.

[63]  Morten H. Christiansen,et al.  Hierarchical and sequential processing of language , 2018, Language, Cognition and Neuroscience.

[64]  R. Levy.  Expectation-based syntactic comprehension , 2008, Cognition.

[65]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[66]  Yonghui Wu,et al.  Exploring the Limits of Language Modeling , 2016, ArXiv.

[67]  Richard L. Lewis,et al.  An Activation-Based Model of Sentence Processing as Skilled Memory Retrieval , 2005, Cogn. Sci..

[68]  A. Nobre,et al.  The Response of Left Temporal Cortex to Sentences , 2002, Journal of Cognitive Neuroscience.

[69]  J. Dotlacil.  Parsing as a Cue‐Based Retrieval Model , 2021, Cogn. Sci..

[70]  Stephani Foraker,et al.  Memory structures that subserve sentence comprehension , 2003 .

[71]  Evelina Fedorenko,et al.  Domain-General Brain Regions Do Not Track Linguistic Input as Closely as Language-Selective Regions , 2017, The Journal of Neuroscience.

[72]  Philip Resnik,et al.  Left-Corner Parsing and Psychological Plausibility , 1992, COLING.

[73]  John M. Henderson,et al.  Language structure in the brain: A fixation-related fMRI study of syntactic surprisal in reading , 2016, NeuroImage.

[74]  M. Jung,et al.  Prefrontal cortex and hippocampus subserve different components of working memory in rats. , 2008, Learning & memory.

[75]  J. S. Evans,et al.  Understanding sentences with relative clauses , 1976, Memory & cognition.

[76]  S. Frank,et al.  Insensitivity of the Human Sentence-Processing System to Hierarchical Structure , 2011, Psychological science.

[77]  A. A. Wijers,et al.  Localizing components of a complex task: sentence processing and working memory , 1998, Neuroreport.

[78]  J. Trueswell,et al.  Cognitive control and parsing: Reexamining the role of Broca’s area in sentence comprehension , 2005, Cognitive, affective & behavioral neuroscience.

[79]  William Schuler,et al.  Surprisal Estimators for Human Reading Times Need Character Models , 2021, ACL.

[80]  John Hale,et al.  A Probabilistic Earley Parser as a Psycholinguistic Model , 2001, NAACL.

[81]  Continuous-time deconvolutional regression for psycholinguistic modeling , 2021, Cognition.

[82]  William Schuler,et al.  Left-Corner Parsing With Distributed Associative Memory Produces Surprisal and Locality Effects. , 2018, Cognitive science.

[83]  Richard Futrell,et al.  The Natural Stories corpus: a reading-time corpus of English texts containing rare syntactic constructions , 2020, Language Resources and Evaluation.

[84]  Liina Pylkkänen,et al.  Simple Composition: A Magnetoencephalography Investigation into the Comprehension of Minimal Linguistic Phrases , 2011, The Journal of Neuroscience.

[85]  Randall Hendrick,et al.  Memory-Load Interference in Syntactic Processing , 2002, Psychological science.

[86]  Edward Gibson,et al.  The nature of working memory capacity in sentence comprehension : Evidence against domain-specific working memory resources , 2006 .

[87]  T. Robbins,et al.  Planning and spatial working memory following frontal lobe lesions in man , 1990, Neuropsychologia.

[88]  D. Weinberger,et al.  Neonatal Damage of the Ventral Hippocampus Impairs Working Memory in the Rat , 2002, Neuropsychopharmacology.

[89]  M. Farah,et al.  A unified account of cognitive impairments following frontal lobe damage: the role of working memory in complex, organized behavior. , 1993, Journal of experimental psychology. General.

[90]  A D Friederici,et al.  Syntactic Working Memory and the Establishment of Filler-Gap Dependencies: Insights from ERPs and fMRI , 2001, Journal of psycholinguistic research.

[91]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[92]  Aniruddh D. Patel,et al.  Language, music, syntax and the brain , 2003, Nature Neuroscience.

[93]  Janet D. Fodor,et al.  The sausage machine: A new two-stage parsing model , 1978, Cognition.

[94]  Antal van den Bosch,et al.  Prediction During Natural Language Comprehension. , 2016, Cerebral cortex.

[95]  Leila Wehbe,et al.  Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain) , 2019, NeurIPS.

[96]  Lila Davachi,et al.  Are Representations in Working Memory Distinct From Representations in Long-Term Memory? , 2010, Psychological science.

[97]  S. Frank,et al.  The ERP response to the amount of information conveyed by words in sentences , 2015, Brain and Language.

[98]  Dae-Shik Kim,et al.  Origin of Negative Blood Oxygenation Level—Dependent fMRI Signals , 2002, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[99]  C. Clifton,et al.  Comprehending Sentences with Long-Distance Dependencies , 1989 .

[100]  A. Yonelinas.  The hippocampus supports high-resolution binding in the service of perception, working memory and long-term memory , 2013, Behavioural Brain Research.

[101]  William Schuler,et al.  Deconvolutional Time Series Regression: A Technique for Modeling Temporally Diffuse Effects , 2018, EMNLP.

[102]  L. Nadel,et al.  Hippocampus: cognitive map or working memory? , 1980, Behavioral and neural biology.

[103]  P. Hagoort.  On Broca, brain, and binding: a new framework , 2005, Trends in Cognitive Sciences.

[104]  P. Goldman-Rakic.  Topography of cognition: parallel distributed networks in primate association cortex. , 1988, Annual review of neuroscience.

[105]  G. Handelmann,et al.  Hippocampus, space, and memory , 1979 .

[106]  J. Fell,et al.  Cross-frequency coupling supports multi-item working memory in the human hippocampus , 2010, Proceedings of the National Academy of Sciences.

[107]  Richard L. Lewis.  Interference in short-term memory: The magical number two (or three) in sentence processing , 1996, Journal of psycholinguistic research.

[108]  William Schuler,et al.  An Analysis of Frequency- and Memory-Based Processing Costs , 2013, NAACL.

[109]  Edward Gibson,et al.  The Nature of Working Memory in Linguistic, Arithmetic and Spatial Integration Processes. , 2007 .

[110]  Richard L. Lewis,et al.  Distinguishing effects of structure and decay on attachment and repair: A cue-based parsing account of recovery from misanalyzed ambiguities , 2003 .

[111]  Roel M. Willems,et al.  Grounding the neurobiology of language in first principles: The necessity of non-language-centric explanations for language comprehension , 2018, Cognition.

[112]  Richard L. Lewis,et al.  Argument-Head Distance and Processing Complexity: Explaining both Locality and Antilocality Effects , 2006 .

[113]  J. Duncan.  The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour , 2010, Trends in Cognitive Sciences.

[114]  Nancy Kanwisher,et al.  Functional specificity for high-level linguistic processing in the human brain , 2011, Proceedings of the National Academy of Sciences.

[115]  Evelina Fedorenko,et al.  The Domain-General Multiple Demand (MD) Network Does Not Support Core Aspects of Language Comprehension: A Large-Scale fMRI Investigation , 2020, The Journal of Neuroscience.

[116]  Stanislas Dehaene,et al.  A Temporal Bottleneck in the Language Comprehension Network , 2012, The Journal of Neuroscience.

[117]  Evelina Fedorenko,et al.  Language-selective brain regions track linguistic input more closely than domain-general regions , 2016 .

[118]  Sneha Shashidhara,et al.  Individual-subject Functional Localization Increases Univariate Activation but Not Multivariate Pattern Discriminability in the “Multiple-demand” Frontoparietal Network , 2020, Journal of Cognitive Neuroscience.

[119]  G. Hickok,et al.  Recency preference in the human sentence processing mechanism , 1996, Cognition.

[120]  N. Kanwisher,et al.  A functional dissociation between language and multiple-demand systems revealed in patterns of BOLD signal fluctuations. , 2014, Journal of neurophysiology.

[121]  D. Heeger,et al.  Linear Systems Analysis of Functional Magnetic Resonance Imaging in Human V1 , 1996, The Journal of Neuroscience.

[122]  Evelina Fedorenko,et al.  Syntactic processing is distributed across the language system , 2016, NeuroImage.

[123]  Edward Gibson,et al.  Consequences of the Serial Nature of Linguistic Input for Sentenial Complexity , 2005, Cogn. Sci..

[124]  Antal van den Bosch,et al.  Distinguishing Syntactic Operations in the Brain: Dependency and Phrase-Structure Parsing , 2020, Neurobiology of Language.

[125]  Nathaniel J. Smith,et al.  The effect of word predictability on reading time is logarithmic , 2013, Cognition.

[126]  Mark D'Esposito,et al.  Variation of BOLD hemodynamic responses across subjects and brain regions and their effects on statistical analyses , 2004, NeuroImage.

[127]  M. Gazzaniga,et al.  Cognitive Neuroscience: The Biology of the Mind , 1998 .

[128]  N. Logothetis,et al.  Negative functional MRI response correlates with decreases in neuronal activity in monkey visual area V1 , 2006, Nature Neuroscience.

[129]  William Schuler,et al.  Hierarchic syntax improves reading time prediction , 2015, NAACL.

[130]  M. Kutas,et al.  Who Did What and When? Using Word- and Clause-Level ERPs to Monitor Working Memory Usage in Reading , 1995, Journal of Cognitive Neuroscience.

[131]  D. Shankweiler,et al.  Unification of sentence processing via ear and eye: An fMRI study , 2011, Cortex.

[132]  Evelina Fedorenko,et al.  Functionally distinct language and Theory of Mind networks are synchronized at rest and during language comprehension. , 2019, Journal of neurophysiology.

[133]  Angela R. Laird,et al.  Modelling neural correlates of working memory: A coordinate-based meta-analysis , 2012, NeuroImage.

[134]  Rainer Goebel,et al.  Measuring structural–functional correspondence: Spatial variability of specialised brain regions after macro-anatomical alignment , 2012, NeuroImage.

[135]  Frank Keller,et al.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity , 2008, Cognition.

[136]  S. Dehaene,et al.  Cortical representation of the constituent structure of sentences , 2011, Proceedings of the National Academy of Sciences.

[137]  Fernanda Ferreira,et al.  Integration and Prediction in Language Processing: A Synthesis of Old and New , 2018, Current directions in psychological science.

[138]  A. Baddeley,et al.  Amnesia and the distinction between long- and short-term memory. , 1970 .

[139]  B. McElree,et al.  Retrieval interference in sentence comprehension. , 2006, Journal of memory and language.

[140]  J. A. Camilleri,et al.  Definition and characterization of an extended multiple-demand network , 2018, NeuroImage.

[141]  A. Anwander,et al.  The brain differentiates human and non-human grammars: Functional localization and structural connectivity , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[142]  Evelina Fedorenko,et al.  The syntactic complexity of Russian relative clauses , 2012, Journal of memory and language.

[143]  E. Bates,et al.  INDIVIDUAL DIFFERENCES AND THEIR IMPLICATIONS FOR THEORIES OF LANGUAGE DEVELOPMENT , 1995 .