From the SelectedWorks of Marcel Adam Just 2011 Commonality of neural representations of words and pictures

In this work we explore whether the patterns of brain activity associated with thinking about concrete objects are dependent on stimulus presentation format, whether an object is referred to by a written or pictorial form. Multi-voxel pattern analysis methods were applied to brain imaging (fMRI) data to identify the item category associated with brief viewings of each of 10 words (naming 5 tools and 5 dwellings) and, separately, with brief viewings of each of 10 pictures (line drawings) of the objects named by the words. These methods were able to identify the category of the picture the participant was viewing, based on neural activation patterns observed during word-viewing, and identify the category of the word the participant was viewing, based on neural activation patterns observed during picture-viewing, using data from only that participant or only from other participants. These results provide an empirical demonstration of object category identification across stimulus formats and across participants. In addition, we were able to identify the category of the word that the participant was viewing based on the patterns of neural activation generated during word-viewing by that participant or by all other participants. Similarly, we were able to identify with even higher accuracy the category of the picture the participant was viewing, based on the patterns of neural activation demonstrated during picture-viewing by that participant or by all other participants. The brain locations that were important for category identification were similar across participants and were distributed throughout the cortex where various object properties might be neurally represented. These findings indicate consistent triggering of semantic representations using different stimulus formats and suggest the presence of stable, distributed, and identifiable neural states that are common to pictorial and verbal input referring to object categories.

[1]  Friedemann Pulvermüller,et al.  Brain mechanisms linking language and action , 2005, Nature Reviews Neuroscience.

[2]  Lisa Aziz-Zadeh,et al.  Embodied semantics for actions: Findings from functional brain imaging , 2008, Journal of Physiology-Paris.

[3]  Larry Gates,et al.  Distinct and shared cortical regions of the human brain activated by pictorial depictions versus verbal descriptions: an fMRI study , 2005, NeuroImage.

[4]  Dinggang Shen,et al.  Classifying spatial patterns of brain activity with machine learning methods: Application to lie detection , 2005, NeuroImage.

[5]  Emiliano Ricciardi,et al.  Beyond sensory images: Object-based representation in the human ventral pathway. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[6]  J. Haxby,et al.  Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects , 1999, Nature Neuroscience.

[7]  R. Raizada,et al.  Quantifying the adequacy of neural representations for a cross-language phonetic discrimination task: prediction of individual differences. , 2010, Cerebral cortex.

[8]  Karalyn Patterson,et al.  Semantic memory disorders , 1997, Trends in Cognitive Sciences.

[9]  P. Matthews,et al.  Category-related activation for written words in the posterior fusiform is task specific , 2005, Neuropsychologia.

[10]  Bertrand Thirion,et al.  Deciphering Cortical Number Coding from Human Brain Activity Patterns , 2009, Current Biology.

[11]  Stephen José Hanson,et al.  Combinatorial codes in ventral temporal lobe for object recognition: Haxby (2001) revisited: is there a “face” area? , 2004, NeuroImage.

[12]  V. Michel,et al.  Recruitment of an Area Involved in Eye Movements During Mental Arithmetic , 2009, Science.

[13]  Uta Frith,et al.  Theory of mind , 2001, Current Biology.

[14]  Vaidehi S. Natu,et al.  Category-Specific Cortical Activity Precedes Retrieval During Memory Search , 2005, Science.

[15]  S. Edelman,et al.  Differential Processing of Objects under Various Viewing Conditions in the Human Lateral Occipital Complex , 1999, Neuron.

[16]  Karl J. Friston,et al.  Where bottom-up meets top-down: neuronal interactions during perception and imagery. , 2004, Cerebral cortex.

[17]  R. Henson,et al.  Multiple levels of visual object constancy revealed by event-related fMRI of repetition priming , 2002, Nature Neuroscience.

[18]  A. Glenberg,et al.  What memory is for: Creating meaning in the service of action , 1997, Behavioral and Brain Sciences.

[19]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[20]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[21]  Alice J. O'Toole,et al.  Partially Distributed Representations of Objects and Faces in Ventral Temporal Cortex , 2005, Journal of Cognitive Neuroscience.

[22]  C. Frith,et al.  Reading the mind in cartoons and stories: an fMRI study of ‘theory of mind’ in verbal and nonverbal tasks , 2000, Neuropsychologia.

[23]  Tom Michael Mitchell,et al.  From the SelectedWorks of Marcel Adam Just 2008 Using fMRI brain activation to identify cognitive states associated with perception of tools and dwellings , 2016 .

[24]  Yaroslav O. Halchenko,et al.  Brain Reading Using Full Brain Support Vector Machines for Object Recognition: There Is No Face Identification Area , 2008, Neural Computation.

[25]  Roel M. Willems,et al.  Seeing and Hearing Meaning: ERP and fMRI Evidence of Word versus Picture Integration into a Sentence Context , 2008, Journal of Cognitive Neuroscience.

[26]  T. Carlson,et al.  Patterns of Activity in the Categorical Representations of Objects , 2003 .

[27]  Richard S. J. Frackowiak,et al.  Functional anatomy of a common semantic system for words and pictures , 1996, Nature.

[28]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[29]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[30]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[31]  Ravi S. Menon,et al.  Differential Effects of Viewpoint on Object-Driven Activation in Dorsal and Ventral Streams , 2002, Neuron.

[32]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[33]  Ferath Kherif,et al.  Distributed cell assemblies for general lexical and category‐specific semantic processing as revealed by fMRI cluster analysis , 2009, Human brain mapping.

[34]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[35]  A. Galaburda,et al.  Human Cerebral Cortex: Localization, Parcellation, and Morphometry with Magnetic Resonance Imaging , 1992, Journal of Cognitive Neuroscience.

[36]  Stephen José Hanson,et al.  Decoding the Large-Scale Structure of Brain Function by Classifying Mental States Across Individuals , 2009, Psychological science.

[37]  Tom Michael Mitchell,et al.  A Neurosemantic Theory of Concrete Noun Representation Based on the Underlying Brain Codes , 2010, PloS one.

[38]  Tom M. Mitchell,et al.  Machine learning classifiers and fMRI: A tutorial overview , 2009, NeuroImage.

[39]  L. Tyler,et al.  Unitary vs multiple semantics: PET studies of word and picture processing , 2004, Brain and Language.

[40]  L. Buxbaum,et al.  Distinctions between manipulation and function knowledge of objects: evidence from functional magnetic resonance imaging. , 2005, Brain research. Cognitive brain research.

[41]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[42]  Janaina Mourão Miranda,et al.  Classifying brain states and determining the discriminating activation patterns: Support Vector Machine on functional MRI data , 2005, NeuroImage.

[43]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[44]  Christian Keysers,et al.  Testing Simulation Theory with Cross-Modal Multivariate Classification of fMRI Data , 2008, PloS one.

[45]  Stefano F. Cappa,et al.  Word and picture matching: a PET study of semantic category effects , 1999, Neuropsychologia.

[46]  Markus Kiefer,et al.  Conceptual Flexibility in the Human Brain: Dynamic Recruitment of Semantic Maps from Visual, Motor, and Motion-related Areas , 2008, Journal of Cognitive Neuroscience.

[47]  M. Chee,et al.  Overlap and Dissociation of Semantic Processing of Chinese Characters, English Words, and Pictures: Evidence from fMRI , 2000, NeuroImage.

[48]  Alice J. O'Toole,et al.  Theoretical, Statistical, and Practical Perspectives on Pattern-based Classification Approaches to the Analysis of Functional Neuroimaging Data , 2007, Journal of Cognitive Neuroscience.

[49]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.