Commonality of neural representations of sentences across languages: Predicting brain activation during Portuguese sentence comprehension using an English-based model of brain function

Abstract The aim of the study was to test the cross‐language generative capability of a model that predicts neural activation patterns evoked by sentence reading, based on a semantic characterization of the sentence. In a previous study on English monolingual speakers (Wang et al., submitted), a computational model performed a mapping from a set of 42 concept‐level semantic features (Neurally Plausible Semantic Features, NPSFs) as well as 6 thematic role markers to neural activation patterns (assessed with fMRI), to predict activation levels in a network of brain locations. The model used two types of information gained from the English‐based fMRI data to predict the activation for individual sentences in Portuguese. First, it used the mapping weights from NPSFs to voxel activation levels derived from the model for English reading. Second, the brain locations for which the activation levels were predicted were derived from a factor analysis of the brain activation patterns during English reading. These meta‐language locations were defined by the clusters of voxels with high loadings on each of the four main dimensions (factors), namely people, places, actions and feelings, underlying the neural representations of the stimulus sentences. This cross‐language model succeeded in predicting the brain activation patterns associated with the reading of 60 individual Portuguese sentences that were entirely new to the model, attaining accuracies reliably above chance level. The prediction accuracy was not affected by whether the Portuguese speaker was monolingual or Portuguese‐English bilingual. The model's confusion errors indicated an accurate capture of the events or states described in the sentence at a conceptual level. Overall, the cross‐language predictive capability of the model demonstrates the neural commonality between speakers of different languages in the representations of everyday events and states, and provides an initial characterization of the common meta‐language neural basis. HighlightsAn English‐based model predicted the fMRI patterns for Portuguese sentences.A cross‐language model predicted both monolinguals’ and bilinguals’ activation.Sentences from two languages were decodable from the same set of 38 brain locations.

[1]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.

[2]  Marcel Adam Just,et al.  Neural Representations of Physics Concepts , 2016, Psychological science.

[3]  Elliot T. Berkman,et al.  Neural Systems Underlying the Reappraisal of Personally Craved Foods , 2014, Journal of Cognitive Neuroscience.

[4]  Albert Costa,et al.  Do bilinguals show neural differences with monolinguals when processing their native language? , 2015, Brain and Language.

[5]  Ioulia Kovelman,et al.  Bilingual and monolingual brains compared: A functional magnetic resonance imaging investigation of syntactic processing and a possible “neural signature” of bilingualism , 2008 .

[6]  Miles Osborne,et al.  Statistical Machine Translation , 2010, Encyclopedia of Machine Learning and Data Mining.

[7]  Leslie G. Ungerleider,et al.  Neural correlates of category-specific knowledge , 1996, Nature.

[8]  M. Seghier,et al.  Where, When and Why Brain Activation Differs for Bilinguals and Monolinguals during Picture Naming and Reading Aloud , 2011, Cerebral cortex.

[9]  Tom M. Mitchell,et al.  Selecting Corpus-Semantic Models for Neurolinguistic Decoding , 2012, *SEMEVAL.

[10]  Mark A. Chevillet,et al.  Evaluating semantic models with word-sentence relatedness , 2016, ArXiv.

[11]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[12]  Albert Costa,et al.  How does the bilingual experience sculpt the brain? , 2014, Nature Reviews Neuroscience.

[13]  Peter Wiemer-Hastings,et al.  Latent semantic analysis , 2004, Annu. Rev. Inf. Sci. Technol..

[14]  Tom Michael Mitchell,et al.  A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes , 2010, PloS one.

[15]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[16]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[17]  F. Pulvermüller,et al.  Nouns, verbs, objects, actions, and abstractions: Local fMRI activity indexes semantics, not lexical categories , 2014, Brain and Language.

[18]  Steven M Frankland,et al.  An architecture for encoding sentence meaning in left mid-superior temporal cortex , 2015, Proceedings of the National Academy of Sciences.

[19]  M. Kronbichler,et al.  Words in Context: The Effects of Length, Frequency, and Predictability on Brain Responses During Natural Reading , 2016, Cerebral cortex.

[20]  Ioulia Kovelman,et al.  Bilingual and Monolingual Brains Compared: A Functional Magnetic Resonance Imaging Investigation of Syntactic Processing and a Possible Neural Signature of Bilingualism , 2008, Journal of Cognitive Neuroscience.

[21]  A. Schwartz,et al.  High-performance neuroprosthetic control by an individual with tetraplegia , 2013, The Lancet.

[22]  Jin Fan,et al.  Neural basis of cultural influence on self-representation , 2007, NeuroImage.

[23]  Murray Grossman,et al.  Causal Evidence for a Mechanism of Semantic Integration in the Angular Gyrus as Revealed by High-Definition Transcranial Direct Current Stimulation , 2016, The Journal of Neuroscience.

[24]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[25]  Indrayana Rustandi,et al.  Integrating Multiple-Study Multiple-Subject fMRI Datasets Using Canonical Correlation Analysis , 2009 .

[26]  Christian Büchel,et al.  Neural circuitry of the bilingual mental lexicon: Effect of age of second language acquisition , 2010, Brain and Cognition.

[27]  Ping Li,et al.  Disentangling narrow and coarse semantic networks in the brain: The role of computational models of word meaning , 2017, Behavior research methods.

[28]  M. Just,et al.  Identifying Emotions on the Basis of Neural Activation , 2013, PloS one.

[29]  Sven Haller,et al.  The age of second language acquisition determines the variability in activation elicited by narration in three languages in Broca’s and Wernicke’s area , 2009, Neuropsychologia.

[30]  Max A. Viergever,et al.  The first taste is always with the eyes: A meta-analysis on the neural correlates of processing visual food cues , 2011, NeuroImage.

[31]  G. Northoff,et al.  Culture-sensitive neural substrates of human cognition: a transcultural neuroimaging approach , 2008, Nature Reviews Neuroscience.

[32]  Benjamin D. Zinszer,et al.  Semantic Structural Alignment of Neural Representational Spaces Enables Translation between English and Chinese Words , 2016, Journal of Cognitive Neuroscience.

[33]  Frank Van Overwalle,et al.  Distinct neural correlates of social categories and personality traits , 2015, NeuroImage.

[34]  Ashutosh Kumar Singh,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2010 .

[35]  G. Fink,et al.  Being with virtual others: Neural correlates of social interaction , 2006, Neuropsychologia.

[36]  Scott H. Johnson-Frey The neural bases of complex tool use in humans , 2004, Trends in Cognitive Sciences.

[37]  James K Rilling,et al.  The neural correlates of theory of mind within interpersonal interactions , 2004, NeuroImage.

[38]  L. Schilbach,et al.  The Neural Correlates of Social Cognition and Social Interaction , 2015 .

[39]  Tom M. Mitchell,et al.  Identifying bilingual semantic neural representations across languages , 2012, Brain and Language.

[40]  Jack L. Gallant,et al.  A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain , 2012, Neuron.

[41]  Thomas L. Dean,et al.  The atoms of neural computation , 2014, Science.

[42]  Tom M. Mitchell,et al.  Identifying Autism from Neural Representations of Social Interactions: Neurocognitive Markers of Autism , 2014, PloS one.

[43]  Giancarlo Valente,et al.  Brain-Based Translation: fMRI Decoding of Spoken Words in Bilinguals Reveals Language-Independent Semantic Representations in Anterior Temporal Lobe , 2014, The Journal of Neuroscience.

[44]  A. Proverbio,et al.  Electrophysiological Measures of Language Processing in Bilinguals , 2002, Journal of Cognitive Neuroscience.

[45]  Augusto Buchweitz,et al.  The bilingual brain: flexibility and control in the human cortex. , 2013, Physics of life reviews.

[46]  R. Andersen,et al.  Decoding motor imagery from the posterior parietal cortex of a tetraplegic human , 2015, Science.

[47]  R. Adolphs,et al.  NEURAL CORRELATES OF CONCEPTUAL KNOWLEDGE FOR ACTIONS , 2003, Cognitive neuropsychology.