The sensory-motor theory of semantics: Evidence from functional imaging

Abstract This review discusses the contributions of functional imaging (fMRI/PET) to our understanding of how semantic concepts are represented and processed in the human brain. The sensory-motor theory of semantic memory suggests that semantic processing relies on reactivation of sensory-motor representations that were involved in perception and action. More specifically, it attributes an apparent category-specific (e.g. tool vs. animals) organization of semantics to anatomical segregation for different semantic features (e.g. action vs. visual). Within this framework, we will review functional imaging evidence that semantic processing of tools and actions may rely on activations within the visuo-motor system.

[1]  B. Argall,et al.  Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus , 2004, Neuron.

[2]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[3]  J. Hodges,et al.  Charting the progression in semantic dementia: implications for the organisation of semantic memory. , 1995 .

[4]  G. Humphreys,et al.  Hierarchies, similarity, and interactivity in object recognition: “Category-specific” neuropsychological deficits , 2001, Behavioral and Brain Sciences.

[5]  Alex Martin,et al.  Representation of Manipulable Man-Made Objects in the Dorsal Stream , 2000, NeuroImage.

[6]  Vittorio Gallese,et al.  Listening to Action-related Sentences Activates Fronto-parietal Motor Circuits , 2005, Journal of Cognitive Neuroscience.

[7]  Myrna F. Schwartz,et al.  Function and manipulation tool knowledge in apraxia: Knowing ‘what for’ but not ‘how’ , 2000 .

[8]  L. Tyler,et al.  Grammatical categories in the brain: the role of morphological structure. , 2007, Cerebral cortex.

[9]  Isabel Gauthier,et al.  Auditory and Action Semantic Features Activate Sensory-Specific Perceptual Brain Regions , 2003, Current Biology.

[10]  Karl J. Friston,et al.  Generative models, brain function and neuroimaging. , 2001, Scandinavian journal of psychology.

[11]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[12]  Christine D. Wilson,et al.  Grounding conceptual knowledge in modality-specific systems , 2003, Trends in Cognitive Sciences.

[13]  Alex Martin,et al.  Semantic memory and the brain: structure and processes , 2001, Current Opinion in Neurobiology.

[14]  Friedemann Pulvermüller,et al.  Brain mechanisms linking language and action , 2005, Nature Reviews Neuroscience.

[15]  I Law,et al.  Categorization and category effects in normal object recognition A PET Study , 2000, Neuropsychologia.

[16]  Karl J. Friston,et al.  Two distinct neural mechanisms for category-selective responses. , 2006, Cerebral cortex.

[17]  L. Tyler,et al.  Towards a distributed account of conceptual knowledge , 2001, Trends in Cognitive Sciences.

[18]  J. Hodges,et al.  Generating ‘tiger’ as an animal name or a word beginning with T: differences in brain activation , 1996, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[19]  Tim Shallice,et al.  The different neural correlates of action and functional knowledge in semantic memory: an FMRI study. , 2008, Cerebral cortex.

[20]  D. Perani,et al.  The neural correlates of verb and noun processing. A PET study. , 1999, Brain : a journal of neurology.

[21]  M Poncet,et al.  The role of sensorimotor experience in object recognition. A case of multimodal agnosia. , 1991, Brain : a journal of neurology.

[22]  T. Shallice,et al.  Category specific semantic impairments , 1984 .

[23]  M. Rushworth,et al.  The left parietal and premotor cortices: motor attention and selection , 2003, NeuroImage.

[24]  Hanna Damasio,et al.  Premotor and Prefrontal Correlates of Category-Related Lexical Retrieval , 1998, NeuroImage.

[25]  D. Neary,et al.  Semantic dementia: a form of circumscribed cerebral atrophy , 1995 .

[26]  J. Haxby,et al.  Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects , 1999, Nature Neuroscience.

[27]  G. Orban,et al.  Observing Others: Multiple Action Representation in the Frontal Lobe , 2005, Science.

[28]  Martha J Farah,et al.  The Living/Nonliving Dissociation is Not an Artifact: Giving an A Priori Implausible Hypothesis a Strong Test. , 1996, Cognitive neuropsychology.

[29]  G. Rizzolatti,et al.  Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons , 2002, Science.

[30]  M. L. Lambon Ralph,et al.  The role of conceptual knowledge in object use evidence from semantic dementia. , 2000, Brain : a journal of neurology.

[31]  G. Rizzolatti,et al.  The Cortical Motor System , 2001, Neuron.

[32]  T. Shallice,et al.  Category specific semantic impairments. , 1998, Brain : a journal of neurology.

[33]  Leslie G. Ungerleider,et al.  Discrete Cortical Regions Associated with Knowledge of Color and Knowledge of Action , 1995, Science.

[34]  Karl J. Friston,et al.  The effect of prior visual information on recognition of speech and sounds. , 2008, Cerebral cortex.

[35]  A. Roepstorff,et al.  Motion verb sentences activate left posterior middle temporal cortex despite static context , 2005, Neuroreport.

[36]  G. Rizzolatti,et al.  I Know What You Are Doing A Neurophysiological Study , 2001, Neuron.

[37]  J R Hodges,et al.  "What" and "how": evidence for the dissociation of object knowledge and mechanical problem-solving skills in the human brain. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[38]  Scott T. Grafton,et al.  Premotor Cortex Activation during Observation and Naming of Familiar Tools , 1997, NeuroImage.

[39]  L. K. Tyler,et al.  Conceptual Structure and the Structure of Concepts: A Distributed Account of Category-Specific Deficits , 2000, Brain and Language.

[40]  J. Hodges,et al.  Non-verbal semantic impairment in semantic dementia , 2000, Neuropsychologia.

[41]  Karl J. Friston,et al.  Effects of visual deprivation on the organization of the semantic system. , 2003, Brain : a journal of neurology.

[42]  Olaf B. Paulson,et al.  The role of action knowledge in the comprehension of artefacts— A PET study , 2000, NeuroImage.

[43]  C. Price,et al.  Functional imaging of the semantic system: Retrieval of sensory-experienced and verbally learned knowledge , 2003, Brain and Language.

[44]  A. Caramazza,et al.  Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction , 1998, Journal of Cognitive Neuroscience.

[45]  G. Rizzolatti,et al.  Neural Circuits Underlying Imitation Learning of Hand Actions An Event-Related fMRI Study , 2004, Neuron.

[46]  Olaf Hauk,et al.  Experience-dependent Plasticity of Conceptual Representations in Human Sensory-Motor Areas , 2007, Journal of Cognitive Neuroscience.

[47]  Leslie G. Ungerleider,et al.  Neural correlates of category-specific knowledge , 1996, Nature.

[48]  Karl J. Friston,et al.  A Dynamic Causal Modeling Study on Category Effects: BottomUp or TopDown Mediation? , 2003, Journal of Cognitive Neuroscience.

[49]  G. Rizzolatti,et al.  Localization of grasp representations in humans by PET: 1. Observation versus execution , 1996, Experimental Brain Research.

[50]  M. Jeannerod,et al.  Mental imaging of motor activity in humans , 1999, Current Opinion in Neurobiology.

[51]  Alan C. Evans,et al.  Specific Involvement of Human Parietal Systems and the Amygdala in the Perception of Biological Motion , 1996, The Journal of Neuroscience.

[52]  G. Rizzolatti,et al.  Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study , 2001, The European journal of neuroscience.

[53]  Karl J. Friston,et al.  Human Brain Function , 1997 .

[54]  R Job,et al.  Category-Specific Naming Impairments? Yes , 1993, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[55]  H. Sakata,et al.  Selectivity for the shape, size, and orientation of objects for grasping in neurons of monkey parietal area AIP. , 2000, Journal of neurophysiology.

[56]  J. Haxby,et al.  Parallel Visual Motion Processing Streams for Manipulable Objects and Human Movements , 2002, Neuron.

[57]  C. Price,et al.  A functional neuroimaging study of the variables that generate category-specific object processing differences. , 1999, Brain : a journal of neurology.

[58]  E. Bullmore,et al.  The functional neuroanatomy of implicit-motion perception or ‘representational momentum’ , 2000, Current Biology.

[59]  M. Arbib,et al.  Language within our grasp , 1998, Trends in Neurosciences.

[60]  N. A. Borghese,et al.  Different Brain Correlates for Watching Real and Virtual Hand Actions , 2001, NeuroImage.

[61]  Glyn W. Humphreys,et al.  Semantic systems or system? Neuropsychological evidence re-examined , 1988 .

[62]  J. Hodges,et al.  Charting the progression in semantic dementia: implications for the organisation of semantic memory. , 1995, Memory.

[63]  G. Rizzolatti,et al.  Neural Circuits Involved in the Recognition of Actions Performed by Nonconspecifics: An fMRI Study , 2004, Journal of Cognitive Neuroscience.

[64]  S. Petersen,et al.  PET activation of posterior temporal regions during auditory word presentation and verb generation. , 1996, Cerebral cortex.

[65]  M. Petrides,et al.  Orofacial somatomotor responses in the macaque monkey homologue of Broca's area , 2005, Nature.

[66]  L. Buxbaum,et al.  Distinctions between manipulation and function knowledge of objects: evidence from functional magnetic resonance imaging. , 2005, Brain research. Cognitive brain research.

[67]  Olaf B. Paulson,et al.  When Action Turns into Words. Activation of Motor-Based Knowledge during Categorization of Manipulable Objects , 2002, Journal of Cognitive Neuroscience.

[68]  Karl J. Friston,et al.  Action selectivity in parietal and temporal cortex. , 2005, Brain research. Cognitive brain research.

[69]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[70]  J. Mazziotta,et al.  Cortical mechanisms of human imitation. , 1999, Science.

[71]  E. Warrington,et al.  Categories of knowledge. Further fractionations and an attempted integration. , 1987, Brain : a journal of neurology.

[72]  F. Fang,et al.  Cortical responses to invisible objects in the human dorsal and ventral pathways , 2005, Nature Neuroscience.

[73]  M. Brett,et al.  Actions Speak Louder Than Functions: The Importance of Manipulability and Action in Tool Representation , 2003, Journal of Cognitive Neuroscience.

[74]  D. Perani,et al.  The Effects of Semantic Category and Knowledge Type on Lexical-Semantic Access: A PET Study , 1998, NeuroImage.

[75]  Alfonso Caramazza,et al.  Cortical signatures of noun and verb production , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[76]  James L. McClelland,et al.  Structure and deterioration of semantic memory: a neuropsychological and computational investigation. , 2004, Psychological review.

[77]  P. Fletcher,et al.  Neural processing of nouns and verbs: the role of inflectional morphology , 2004, Neuropsychologia.

[78]  Guido Gainotti,et al.  What the Locus of Brain Lesion Tells us About the Nature of the Cognitive Defect Underlying Category-Specific Disorders: A Review , 2000, Cortex.

[79]  G. Rizzolatti,et al.  Parietal Lobe: From Action Organization to Intention Understanding , 2005, Science.

[80]  J. Decety,et al.  Effect of subjective perspective taking during simulation of action: a PET investigation of agency , 2001, Nature Neuroscience.

[81]  G. Humphreys,et al.  Calling a squirrel a squirrel but a canoe a wigwam: a category-specific deficit for artefactual objects and body parts , 1992 .

[82]  Stefano F. Cappa,et al.  Word and picture matching: a PET study of semantic category effects , 1999, Neuropsychologia.

[83]  Alex Martin,et al.  Experience-dependent modulation of category-related cortical activity. , 2002, Cerebral cortex.

[84]  A. Damasio,et al.  A neural basis for lexical retrieval , 1996, Nature.

[85]  Karl J. Friston,et al.  The Feature-Based Model of Semantic Memory , 2004 .

[86]  Richard J. Epstein,et al.  Current perspectives in dysphasia , 1985 .

[87]  M. Jeannerod Neural Simulation of Action: A Unifying Mechanism for Motor Cognition , 2001, NeuroImage.

[88]  A. Caramazza,et al.  WHAT ARE THE FACTS OF SEMANTIC CATEGORY-SPECIFIC DEFICITS? A CRITICAL REVIEW OF THE CLINICAL EVIDENCE , 2003, Cognitive neuropsychology.

[89]  R. E Passingham,et al.  Activations related to “mirror” and “canonical” neurones in the human brain: an fMRI study , 2003, NeuroImage.

[90]  Leonardo Fogassi,et al.  Motor functions of the parietal lobe , 2005, Current Opinion in Neurobiology.

[91]  Anthony Randal McIntosh,et al.  Towards a network theory of cognition , 2000, Neural Networks.

[92]  T. Rogers,et al.  Where do you know what you know? The representation of semantic knowledge in the human brain , 2007, Nature Reviews Neuroscience.

[93]  Uta Noppeney,et al.  Temporal lobe lesions and semantic impairment: a comparison of herpes simplex virus encephalitis and semantic dementia. , 2006, Brain : a journal of neurology.

[94]  Scott T. Grafton,et al.  Actions or Hand-Object Interactions? Human Inferior Frontal Cortex and Action Observation , 2003, Neuron.

[95]  G. Gainotti,et al.  Neuroanatomical correlates of category-specific semantic disorders: a critical survey. , 1995, Memory.

[96]  L. Tyler,et al.  Binding crossmodal object features in perirhinal cortex. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[97]  Richard S. J. Frackowiak,et al.  Noun and verb retrieval by normal subjects. Studies with PET. , 1996, Brain : a journal of neurology.

[98]  M M Mesulam,et al.  Large‐scale neurocognitive networks and distributed processing for attention, language, and memory , 1990, Annals of neurology.

[99]  Barry Horwitz,et al.  The elusive concept of brain connectivity , 2003, NeuroImage.

[100]  A. Caramazza,et al.  Category-specific naming and comprehension impairment: a double dissociation. , 1991, Brain : a journal of neurology.

[101]  E. DeYoe,et al.  Distinct Cortical Pathways for Processing Tool versus Animal Sounds , 2005, The Journal of Neuroscience.

[102]  G. Humphreys,et al.  Calling a squirrel a squirrel but a canoe a wigwam: a category-specific deficit for artefactual objects and body parts. , 1998 .

[103]  J. Haxby,et al.  fMRI Responses to Video and Point-Light Displays of Moving Humans and Manipulable Objects , 2003, Journal of Cognitive Neuroscience.

[104]  Michael A. Arbib,et al.  Language evolution: neural homologies and neuroinformatics , 2003, Neural Networks.

[105]  J. Mazziotta,et al.  Grasping the Intentions of Others with One's Own Mirror Neuron System , 2005, PLoS biology.

[106]  M. Farah,et al.  A neural basis for category and modality specificity of semantic knowledge , 1999, Neuropsychologia.

[107]  G. Rizzolatti,et al.  Motor and cognitive functions of the ventral premotor cortex , 2002, Current Opinion in Neurobiology.

[108]  A. Damasio Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition , 1989, Cognition.

[109]  Karl J. Friston,et al.  Dynamic causal modeling , 2010, Scholarpedia.

[110]  John Hart,et al.  Neural activation during an explicit categorization task: category- or feature-specific effects? , 2002, Brain research. Cognitive brain research.

[111]  E. Procyk,et al.  Brain activity during observation of actions. Influence of action content and subject's strategy. , 1997, Brain : a journal of neurology.

[112]  Richard J. Brown Neuropsychology Mental Structure , 1989 .

[113]  Karl J. Friston,et al.  Anatomic Constraints on Cognitive Theories of Category Specificity , 2002, NeuroImage.

[114]  C. Price,et al.  Functional Neuroanatomy of the Semantic System: Divisible by What? , 1998, Journal of Cognitive Neuroscience.

[115]  E. Maguire,et al.  Differential modulation of a common memory retrieval network revealed by positron emission tomography , 1999, Hippocampus.

[116]  E. Bizzi,et al.  The Cognitive Neurosciences , 1996 .

[117]  Alex Martin,et al.  A neural system for learning about object function. , 2006, Cerebral cortex.

[118]  Guido Gainotti,et al.  Cognitive and anatomical locus of lesion in a patient with a category-specific semantic impairment for living beings , 1996 .

[119]  N. Kanwisher,et al.  Activation in Human MT/MST by Static Images with Implied Motion , 2000, Journal of Cognitive Neuroscience.

[120]  D. V. von Cramon,et al.  Separable neuronal circuitries for manipulable and non-manipulable objects in working memory. , 2002, Cerebral cortex.

[121]  Glyn W. Humphreys,et al.  The neural substrates of action retrieval: An examination of semantic and visual routes to action , 2002 .

[122]  J. Decety,et al.  Functional anatomy of execution, mental simulation, observation, and verb generation of actions: A meta‐analysis , 2001, Human brain mapping.

[123]  J. Decety,et al.  Top down effect of strategy on the perception of human biological motion: a pet investigation. , 1998, Cognitive neuropsychology.

[124]  Uta Noppeney,et al.  Can segregation within the semantic system account for category-specific deficits? , 2002, Brain : a journal of neurology.