An architecture for encoding sentence meaning in left mid-superior temporal cortex

Significance The 18th-century Prussian philosopher Wilhelm von Humbolt famously noted that natural language makes “infinite use of finite means.” By this, he meant that language deploys a finite set of words to express an effectively infinite set of ideas. As the seat of both language and thought, the human brain must be capable of rapidly encoding the multitude of thoughts that a sentence could convey. How does this work? Here, we find evidence supporting a long-standing conjecture of cognitive science: that the human brain encodes the meanings of simple sentences much like a computer, with distinct neural populations representing answers to basic questions of meaning such as “Who did it?” and “To whom was it done?” Human brains flexibly combine the meanings of words to compose structured thoughts. For example, by combining the meanings of “bite,” “dog,” and “man,” we can think about a dog biting a man, or a man biting a dog. Here, in two functional magnetic resonance imaging (fMRI) experiments using multivoxel pattern analysis (MVPA), we identify a region of left mid-superior temporal cortex (lmSTC) that flexibly encodes “who did what to whom” in visually presented sentences. We find that lmSTC represents the current values of abstract semantic variables (“Who did it?” and “To whom was it done?”) in distinct subregions. Experiment 1 first identifies a broad region of lmSTC whose activity patterns (i) facilitate decoding of structure-dependent sentence meaning (“Who did what to whom?”) and (ii) predict affect-related amygdala responses that depend on this information (e.g., “the baby kicked the grandfather” vs. “the grandfather kicked the baby”). Experiment 2 then identifies distinct, but neighboring, subregions of lmSTC whose activity patterns carry information about the identity of the current “agent” (“Who did it?”) and the current “patient” (“To whom was it done?”). These neighboring subregions lie along the upper bank of the superior temporal sulcus and the lateral bank of the superior temporal gyrus, respectively. At a high level, these regions may function like topographically defined data registers, encoding the fluctuating values of abstract semantic variables. This functional architecture, which in key respects resembles that of a classical computer, may play a critical role in enabling humans to flexibly generate complex thoughts.

[1]  David G. Stork,et al.  Pattern Classification , 1973 .

[2]  J. Fodor The Language of Thought , 1980 .

[3]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[4]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[5]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[6]  Geoffrey E. Hinton Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[7]  L. Shastri,et al.  From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony , 1993, Behavioral and Brain Sciences.

[8]  S. Pinker The Language Instinct , 1994 .

[9]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[10]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[11]  C. von der Malsburg The what and why of binding: the modeler's perspective. , 1999, Neuron.

[12]  E. Bullmore,et al.  Common and Distinct Neural Substrates for Pragmatic, Semantic, and Syntactic Processing of Spoken Sentences: An fMRI Study , 2000, Journal of Cognitive Neuroscience.

[13]  Richard S. Busby,et al.  Generalizable Relational Binding from Coarse-coded Distributed Representations , 2001, NIPS.

[14]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[15]  G. Marcus The Algebraic Mind: Integrating Connectionism and Cognitive Science , 2001 .

[16]  A. Nobre,et al.  The Response of Left Temporal Cortex to Sentences , 2002, Journal of Cognitive Neuroscience.

[17]  C. Fiebach,et al.  The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes. , 2003, Cerebral cortex.

[18]  David P Mackinnon,et al.  Confidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods , 2004, Multivariate behavioral research.

[19]  K. Strimmer,et al.  Statistical Applications in Genetics and Molecular Biology A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics , 2011 .

[20]  Joseph E LeDoux,et al.  Contributions of the Amygdala to Emotion Processing: From Animal Models to Human Behavior , 2005, Neuron.

[21]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[22]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[23]  Colin Humphries,et al.  Syntactic and Semantic Modulation of Neural Activity during Auditory Sentence Comprehension , 2006, Journal of Cognitive Neuroscience.

[24]  Anjan Chatterjee,et al.  The Functional Neuroanatomy of Thematic Role and Locative Relational Knowledge , 2007, Journal of Cognitive Neuroscience.

[25]  Brent L. Hughes,et al.  Prefrontal-Subcortical Pathways Mediating Successful Emotion Regulation , 2008, Neuron.

[26]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[27]  Leonidas A A Doumas,et al.  A theory of the discovery and predication of relational concepts. , 2008, Psychological review.

[28]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[29]  Gregory Hickok,et al.  Selective attention to semantic and syntactic features modulates sentence processing networks in anterior temporal cortex. , 2009, Cerebral cortex.

[30]  Luigi Rizzi,et al.  Sentence Syntax and Content in the Human Temporal Lobe: An fMRI Adaptation Study in Auditory and Visual Modalities , 2009, Journal of Cognitive Neuroscience.

[31]  D. Osherson,et al.  An early stage of conceptual combination: Superimposition of constituent concepts in left anterolateral temporal lobe , 2010, Cognitive neuroscience.

[32]  Liina Pylkkänen,et al.  Simple Composition: A Magnetoencephalography Investigation into the Comprehension of Minimal Linguistic Phrases , 2011, The Journal of Neuroscience.

[33]  Daniel N. Osherson,et al.  Evidence for conceptual combination in the left anterior temporal lobe , 2011, NeuroImage.

[34]  S. Dehaene,et al.  Cortical representation of the constituent structure of sentences , 2011, Proceedings of the National Academy of Sciences.

[35]  Karl J. Friston Functional and Effective Connectivity: A Review , 2011, Brain Connect..

[36]  Nancy Kanwisher,et al.  Functional specificity for high-level linguistic processing in the human brain , 2011, Proceedings of the National Academy of Sciences.

[37]  Francisco Pereira,et al.  Information mapping with pattern classifiers: A comparative study , 2011, NeuroImage.

[38]  Yu-Chin Chiu,et al.  Tracking cognitive fluctuations with multivoxel pattern time course (MVPTC) analysis , 2012, Neuropsychologia.

[39]  Sharon L. Thompson-Schill,et al.  Informational connectivity: identifying synchronized discriminability of multi-voxel patterns across the brain , 2013, Front. Hum. Neurosci..

[40]  Jonathan D. Cohen,et al.  Indirection and symbol-like processing in the prefrontal cortex and basal ganglia , 2013, Proceedings of the National Academy of Sciences.

[41]  Angela D. Friederici,et al.  Syntax gradually segregates from semantics in the developing brain , 2014, NeuroImage.

[42]  L. Aravind,et al.  Integration of Word Meaning and World Knowledge in Language Comprehension , 2022 .