Characterizing the Spatiotemporal Neural Representation of Concrete Nouns Across Paradigms

Most of the work investigating the representation of concrete nouns in the brain has focused on the locations that code the information. We present a model to study the contributions of perceptual and semantic features to the neural code representing concepts over time and space. The model is evaluated using magnetoencephalography data from different paradigms and not only corroborates previous findings regarding a distributed code, but provides further details about how the encoding of different subcomponents varies in the space-time spectrum. The model also successfully generalizes to novel concepts that it has never seen during training, which argues for the combination of specific properties in forming the meaning of concrete nouns in the brain. The results across paradigms are in agreement when the main differences among the experiments (namely, the number of repetitions of the stimulus, the task the subjects performed, and the type of stimulus provided) were taken into consideration. More specifically, these results suggest that features specific to the physical properties of the stimuli, such as word length and right-diagonalness, are encoded in posterior regions of the brain in the first hundreds of milliseconds after stimulus onset. Then, properties inherent to the nouns, such as is it alive? and can you pick it up?, are represented in the signal starting at about 250 ms, focusing on more anterior parts of the cortex. The code for these different features was found to be distributed over time and space, and it was common for several regions to simultaneously code for a particular property. Moreover, most anterior regions were found to code for multiple features, and a complex temporal profile could be observed for the majority of properties. For example, some features inherent to the nouns were encoded earlier than others, and the extent of time in which these properties could be decoded varied greatly among them. These findings complement much of the work previously described in the literature, and offer new insights about the temporal aspects of the neural encoding of concrete nouns. This model provides a spatiotemporal signature of the representation of objects in the brain. Paired with data from carefully-designed paradigms, the model is an important tool with which to analyze the commonalities of the neural code across stimulus modalities and tasks performed by the subjects.

[1]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[2]  R. Salmelin Clinical neurophysiology of language: The MEG approach , 2007, Clinical Neurophysiology.

[3]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[4]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[5]  R. Salmelin,et al.  Distinct time courses of word and context comprehension in the left temporal cortex. , 1998, Brain : a journal of neurology.

[6]  D. Cheyne,et al.  Spatiotemporal mapping of cortical activity accompanying voluntary movements using an event‐related beamforming approach , 2006, Human brain mapping.

[7]  R. Ilmoniemi,et al.  Signal-space projection method for separating MEG or EEG into components , 1997, Medical and Biological Engineering and Computing.

[8]  Riitta Salmelin,et al.  Accessing newly learned names and meanings in the native language , 2009, Human brain mapping.

[9]  J. Rodd,et al.  Anteromedial temporal cortex supports fine-grained differentiation among objects. , 2005, Cerebral cortex.

[10]  A. Caramazza,et al.  The organisation of conceptual knowledge in the brain: The future's past and some future directions , 2006, Cognitive neuropsychology.

[11]  Herbert Schriefers,et al.  Is animacy special? ERP correlates of semantic violations and animacy violations in sentence processing , 2011, Brain Research.

[12]  W. Levelt,et al.  The spatial and temporal signatures of word production components , 2004, Cognition.

[13]  M. Kutas,et al.  An electrophysiological analysis of animacy effects in the processing of object relative sentences. , 1999, Psychophysiology.

[14]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[15]  L. Barsalou Grounded cognition. , 2008, Annual review of psychology.

[16]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[17]  Valerie A. Carr,et al.  Spatiotemporal Dynamics of Modality-Specific and Supramodal Word Processing , 2003, Neuron.

[18]  Kara D. Federmeier,et al.  Electrophysiology reveals semantic memory use in language comprehension , 2000, Trends in Cognitive Sciences.

[19]  P. Hagoort The fractionation of spoken language understanding by measuring electrical and magnetic brain signals , 2008, Philosophical Transactions of the Royal Society B: Biological Sciences.

[20]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[21]  Tom M. Mitchell,et al.  Selecting Corpus-Semantic Models for Neurolinguistic Decoding , 2012, *SEMEVAL.

[22]  Heikki Lyytinen,et al.  Cortical Activation during Spoken-Word Segmentation in Nonreading-Impaired and Dyslexic Adults , 2002, The Journal of Neuroscience.

[23]  F. Pulvermüller Brain reflections of words and their meaning , 2001, Trends in Cognitive Sciences.

[24]  Riitta Salmelin,et al.  Cortical differentiation of speech and nonspeech sounds at 100 ms: implications for dyslexia. , 2005, Cerebral cortex.

[25]  R. Hari,et al.  Magnetoencephalography in the study of human somatosensory cortical processing. , 1999, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[26]  Vladimir Vovk,et al.  Kernel Ridge Regression , 2013, Empirical Inference.

[27]  H. Robinson Principles and Procedures of Statistics , 1961 .

[28]  Tom Michael Mitchell,et al.  A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes , 2010, PloS one.

[29]  Carson C. Chow,et al.  Repetition priming and repetition suppression: A case for enhanced efficiency through neural synchronization , 2012, Cognitive neuroscience.

[30]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[31]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[32]  Michael A. Ford,et al.  Can I have a quick word? Early electrophysiological manifestations of psycholinguistic processes revealed by event-related regression analysis of the EEG , 2009, Biological Psychology.

[33]  Tom M. Mitchell,et al.  Identifying bilingual semantic neural representations across languages , 2012, Brain and Language.

[34]  Robert Oostenveld,et al.  Identifying Object Categories from Event-Related EEG: Toward Decoding of Conceptual Representations , 2010, PloS one.

[35]  S. Taulu,et al.  Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements , 2006, Physics in medicine and biology.

[36]  R. Salmelin,et al.  Neural Correlates of Letter-String Length and Lexicality during Reading in a Regular Orthography , 2003, Journal of Cognitive Neuroscience.

[37]  Simon J. Thorpe,et al.  Ultra-rapid object detection with saccadic eye movements: Visual processing speed revisited , 2006, Vision Research.

[38]  Jan Kujala,et al.  Localization of correlated network activity at the cortical level with MEG , 2008, NeuroImage.

[39]  Sydney S. Cash,et al.  Decoding word and category-specific spatiotemporal representations from MEG and EEG , 2011, NeuroImage.

[40]  Jeremy R. Manning,et al.  Spontaneously Reactivated Patterns in Frontal and Temporal Lobe Predict Semantic Clustering during Memory Search , 2012, The Journal of Neuroscience.

[41]  Vaidehi S. Natu,et al.  Category-Specific Cortical Activity Precedes Retrieval During Memory Search , 2005, Science.

[42]  Friedemann Pulvermüller,et al.  Category specificity in the processing of color-related and form-related words: An ERP study , 2006, NeuroImage.

[43]  R. Salmelin,et al.  Spatiotemporal Convergence of Semantic Processing in Reading and Speech Perception , 2009, The Journal of Neuroscience.

[44]  Matthew H. Davis,et al.  Imagery or meaning? Evidence for a semantic origin of category-specific brain activity in metabolic imaging , 2008, The European journal of neuroscience.

[45]  Thomas Witzel,et al.  Spatiotemporal cortical dynamics underlying abstract and concrete word reading , 2007, Human brain mapping.

[46]  M. Tarr,et al.  Learning to see faces and objects , 2003, Trends in Cognitive Sciences.

[47]  M. Kendall Statistical Methods for Research Workers , 1937, Nature.

[48]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[49]  Paul Bloom,et al.  Do 5-month-old infants see humans as material objects? , 2004, Cognition.

[50]  Uri Hasson,et al.  Not Lost in Translation: Neural Responses Shared Across Languages , 2012, The Journal of Neuroscience.

[51]  B. Mesquita,et al.  Adjustment to Chronic Diseases and Terminal Illness Health Psychology : Psychological Adjustment to Chronic Disease , 2006 .

[52]  Alex Martin,et al.  A wavelet-based method for measuring the oscillatory dynamics of resting-state functional connectivity in MEG , 2011, NeuroImage.

[53]  S. Taulu,et al.  Suppression of Interference and Artifacts by the Signal Space Separation Method , 2003, Brain Topography.

[54]  T. Rogers,et al.  Neural basis of category-specific semantic deficits for living things: evidence from semantic dementia, HSVE and a neural network model. , 2006, Brain : a journal of neurology.

[55]  Wei Wang,et al.  Clustering Linear Discriminant Analysis for MEG-Based Brain Computer Interfaces , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[56]  Friedemann Pulvermüller,et al.  Understanding in an instant: Neurophysiological evidence for mechanistic language circuits in the brain , 2009, Brain and Language.

[57]  Olaf Hauk,et al.  Keep it simple: a case for using classical minimum norm estimation in the analysis of EEG and MEG data , 2004, NeuroImage.

[58]  Alex Martin,et al.  Semantic memory and the brain: structure and processes , 2001, Current Opinion in Neurobiology.

[59]  Tom Michael Mitchell,et al.  From the SelectedWorks of Marcel Adam Just 2008 Using fMRI brain activation to identify cognitive states associated with perception of tools and dwellings , 2016 .

[60]  Indrayana Rustandi,et al.  Predictive fMRI analysis for multiple subjects and multiple studies , 2010 .

[61]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[62]  Riitta Salmelin,et al.  Localization of Syntactic and Semantic Brain Responses using Magnetoencephalography , 2007, Journal of Cognitive Neuroscience.

[63]  Peter C. Hansen,et al.  MEG. An introduction to methods , 2010 .

[64]  R. Hari,et al.  Neuromagnetic responses to vowels vs. tones reveal hemispheric lateralization , 2006, Clinical Neurophysiology.

[65]  Bradford Z. Mahon,et al.  What drives the organization of object knowledge in the brain? , 2011, Trends in Cognitive Sciences.

[66]  Riitta Salmelin,et al.  Cortical Effects of Shifting Letter Position in Letter Strings of Varying Length , 2003, Journal of Cognitive Neuroscience.

[67]  Alex Martin,et al.  Long-lasting cortical plasticity in the object naming system , 2000, Nature Neuroscience.

[68]  Geoffrey E. Hinton,et al.  Zero-shot Learning with Semantic Output Codes , 2009, NIPS.

[69]  Leslie G. Ungerleider Two cortical visual systems , 1982 .

[70]  R. Kass,et al.  Decoding and cortical source localization for intended movement direction with MEG. , 2010, Journal of neurophysiology.

[71]  R. Salmelin,et al.  Spatiotemporal interaction between sound form and meaning during spoken word perception. , 2008, Cerebral cortex.

[72]  Bradford Z. Mahon,et al.  Action-Related Properties Shape Object Representations in the Ventral Stream , 2007, Neuron.

[73]  Massimo Poesio,et al.  Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study , 2012, Front. Neuroinform..

[74]  Richard M. Leahy,et al.  Electromagnetic brain mapping , 2001, IEEE Signal Process. Mag..

[75]  Michel Besserve,et al.  Classification methods for ongoing EEG and MEG signals. , 2007, Biological research.

[76]  R. Cabeza,et al.  Imaging Cognition II: An Empirical Review of 275 PET and fMRI Studies , 2000, Journal of Cognitive Neuroscience.

[77]  Riitta Salmelin,et al.  Tracking neural coding of perceptual and semantic features of concrete nouns , 2012, NeuroImage.

[78]  Tom M. Mitchell,et al.  From the SelectedWorks of Marcel Adam Just 2011 Commonality of neural representations of words and pictures , 2016 .

[79]  Patrick Suppes,et al.  Single-Trial Classification of MEG Recordings , 2007, IEEE Transactions on Biomedical Engineering.

[80]  R. Salmelin,et al.  Dynamics of letter string perception in the human occipitotemporal cortex. , 1999, Brain : a journal of neurology.

[81]  Mia Liljeström,et al.  Comparing MEG and fMRI views to naming actions and objects , 2009, NeuroImage.

[82]  R. Ilmoniemi,et al.  Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain , 1993 .

[83]  HighWire Press Philosophical Transactions of the Royal Society of London , 1781, The London Medical Journal.

[84]  H. Renvall,et al.  Functional Magnetic Resonance Imaging Blood Oxygenation Level-Dependent Signal and Magnetoencephalography Evoked Responses Yield Different Neural Functionality in Reading , 2011, The Journal of Neuroscience.

[85]  Jan Kujala,et al.  Neural interactions at the core of phonological and semantic priming of written words. , 2012, Cerebral cortex.

[86]  D. Plaut Graded modality-specific specialisation in semantics: A computational account of optic aphasia , 2002, Cognitive neuropsychology.

[87]  Alain de Cheveigné,et al.  Time-shift denoising source separation , 2010, Journal of Neuroscience Methods.

[88]  F. N. David,et al.  Principles and procedures of statistics. , 1961 .

[89]  D. Westwood,et al.  Manipulability and living/non-living category effects on object identification , 2005, Brain and Cognition.

[90]  Anders M. Dale,et al.  N400-like Magnetoencephalography Responses Modulated by Semantic Context, Word Frequency, and Lexical Class in Sentences , 2002, NeuroImage.

[91]  Friedemann Pulvermüller,et al.  Brain Signatures of Meaning Access in Action Word Recognition , 2005, Journal of Cognitive Neuroscience.

[92]  Tom M. Mitchell,et al.  Machine learning classifiers and fMRI: A tutorial overview , 2009, NeuroImage.

[93]  Indrayana Rustandi,et al.  Integrating Multiple-Study Multiple-Subject fMRI Datasets Using Canonical Correlation Analysis , 2009 .

[94]  P Suppes,et al.  Invariance of brain-wave representations of simple visual images and their names. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[95]  Rodrigo Quian Quiroga,et al.  Nonlinear multivariate analysis of neurophysiological signals , 2005, Progress in Neurobiology.

[96]  T. Gale,et al.  The Effect of ‘Masking’ on Picture Naming , 2002, Cortex.

[97]  R. Ilmoniemi,et al.  Interpreting magnetic fields of the brain: minimum norm estimates , 2006, Medical and Biological Engineering and Computing.

[98]  Bradford Z. Mahon,et al.  Concepts and categories: a cognitive neuropsychological perspective. , 2009, Annual review of psychology.

[99]  C. Price The anatomy of language: a review of 100 fMRI studies published in 2009 , 2010, Annals of the New York Academy of Sciences.

[100]  Tom M. Mitchell,et al.  Decoding Word Semantics from Magnetoencephalography Time Series Transformations , 2012 .

[101]  P. Nunez,et al.  On the Relationship of Synaptic Activity to Macroscopic Measurements: Does Co-Registration of EEG with fMRI Make Sense? , 2004, Brain Topography.