The Role of Sensory and Motor Information in Semantic Representation: A Review

Publisher Summary Embodied theories of cognition propose that simulation is the basis for cognitive representation. Simulation is assumed to use the same sensory–motor systems that are engaged during real experience; when this principle is applied to the representation of linguistic meaning (semantics) theories propose that semantic content is achieved by recreating, usually in weaker form, the sensory and motor information produced when the referent of a word or sentence is actually experienced. Simulations are content-specific; for example, words referring to motion, such as rise and fall, are thought to recruit sensory systems involved in perceiving motion, and words referring to motor actions, such as kick and walk, are thought to recruit the motor systems used for those actions. Therefore, embodied theories of semantic representation focus on semantic content, rather than the structure of the semantic system as a whole, how words are related to one another, or how categories are represented. At a basic level, embodiment extends the non-controversial idea that we learn from experience, so semantics must be grounded in our sensations and actions; however, it appears to be a departure from the commonly accepted view that semantic content is amodal and thus not dependent on sensory–motor information. This chapter reviews different theories of semantic representation placing them on a continuum as regards to their proposals about the role of sensory and motor information. A brief review of behavioral and neuroscientific evidence is then presented and the chapter ends with a discussion of what the implications are for semantic representation.

[1]  D. Plaut Graded modality-specific specialisation in semantics: A computational account of optic aphasia , 2002, Cognitive neuropsychology.

[2]  Vittorio Gallese,et al.  Listening to Action-related Sentences Activates Fronto-parietal Motor Circuits , 2005, Journal of Cognitive Neuroscience.

[3]  G. Humphreys,et al.  Hierarchies, similarity, and interactivity in object recognition: “Category-specific” neuropsychological deficits , 2001, Behavioral and Brain Sciences.

[4]  L. Tyler,et al.  Towards a distributed account of conceptual knowledge , 2001, Trends in Cognitive Sciences.

[5]  A. A. Wijers,et al.  Visual semantic features are activated during the processing of concrete words: event-related potential evidence for perceptual semantic priming. , 2000, Brain research. Cognitive brain research.

[6]  Michael J. Spivey,et al.  Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects , 2001, Psychological research.

[7]  A. Glenberg,et al.  Symbol Grounding and Meaning: A Comparison of High-Dimensional and Embodied Theories of Meaning , 2000 .

[8]  Sergio E. Chaigneau,et al.  THE SIMILARITY-IN-TOPOGRAPHY PRINCIPLE: RECONCILING THEORIES OF CONCEPTUAL DEFICITS , 2003, Cognitive neuropsychology.

[9]  Rolf A. Zwaan The Immersed Experiencer: Toward An Embodied Theory Of Language Comprehension , 2003 .

[10]  R. Adolphs,et al.  Neural systems behind word and concept retrieval , 2004, Cognition.

[11]  Uta Noppeney,et al.  Can segregation within the semantic system account for category-specific deficits? , 2002, Brain : a journal of neurology.

[12]  C. Spence,et al.  The cost of expecting events in the wrong sensory modality , 2001, Perception & psychophysics.

[13]  R. Jackendoff Foundations of Language: Brain, Meaning, Grammar, Evolution , 2002 .

[14]  Daniel C. Richardson,et al.  Motor representations in memory and mental models: Embodiment in cognition , 2001 .

[15]  Michael P. Kaschak,et al.  The Action-Sentence Compatibility Effect: It's All in the Timing , 2006, Cogn. Sci..

[16]  Julie C. Sedivy,et al.  Playing on the typewriter, typing on the piano: manipulation knowledge of objects , 2006, Cognition.

[17]  J. F. Marques,et al.  Specialization and semantic organization: Evidence for multiple semantics linked to sensory modalities , 2006, Memory & cognition.

[18]  Hanna Damasio,et al.  Premotor and Prefrontal Correlates of Category-Related Lexical Retrieval , 1998, NeuroImage.

[19]  Richard D. Hichwa,et al.  A neural basis for lexical retrieval , 1996, Nature.

[20]  M. Tanenhaus,et al.  Circumscribing Referential Domains during Real-Time Language Comprehension , 2002 .

[21]  Anne Cutler,et al.  A theory of lexical access in speech production , 1999, Behavioral and Brain Sciences.

[22]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[23]  Alfonso Caramazza,et al.  All Talk and No Action: A Transcranial Magnetic Stimulation Study of Motor Cortex Activation during Action Word Production , 2004, Journal of Cognitive Neuroscience.

[24]  F Pulvermüller,et al.  The Right Hemisphere’s Role in Action Word Processing: a Double Case Study. , 2001, Neurocase.

[25]  M. Jeannerod Neural Simulation of Action: A Unifying Mechanism for Motor Cognition , 2001, NeuroImage.

[26]  A. Borghi,et al.  Sentence comprehension and action: Effector specific modulation of the motor system , 2007, Brain Research.

[27]  Michael P. Kaschak,et al.  Perception of Auditory Motion Affects Language Processing , 2006, Cogn. Sci..

[28]  Olaf B. Paulson,et al.  When Action Turns into Words. Activation of Motor-Based Knowledge during Categorization of Manipulable Objects , 2002, Journal of Cognitive Neuroscience.

[29]  Antonio R. Damasio,et al.  The Brain Binds Entities and Events by Multiregional Activation from Convergence Zones , 1989, Neural Computation.

[30]  Rolf A. Zwaan,et al.  Effects of negation and situational presence on the accessibility of text information. , 2003, Journal of experimental psychology. Learning, memory, and cognition.

[31]  F. Pulvermüller Brain reflections of words and their meaning , 2001, Trends in Cognitive Sciences.

[32]  R. Klatzky The role of motor representations in semantic sensibility judgments , 1989 .

[33]  A. Caramazza,et al.  Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction , 1998, Journal of Cognitive Neuroscience.

[34]  Tim Shallice,et al.  Multiple semantics: whose confusions? , 1993 .

[35]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[36]  L. Barsalou,et al.  Verifying Different-Modality Properties for Concepts Produces Switching Costs , 2003, Psychological science.

[37]  M. Tanenhaus,et al.  Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition , 2005, Psychonomic bulletin & review.

[38]  G. Hesslow Conscious thought as simulation of behaviour and perception , 2002, Trends in Cognitive Sciences.

[39]  B. Bahrami,et al.  Motion Detection and Motion Verbs , 2007, Psychological science.

[40]  Rolf A. Zwaan,et al.  Language Comprehenders Mentally Represent the Shapes of Objects , 2002, Psychological science.

[41]  A. Damasio,et al.  Cortical systems for retrieval of concrete knowledge: The convergence zone framework , 1994 .

[42]  M. Jeannerod,et al.  Word processing in Parkinson's disease is impaired for action verbs but not for concrete nouns , 2008, Neuropsychologia.

[43]  E. Warrington,et al.  CATEGORY SPECIFIC ACCESS DYSPHASIA , 1983 .

[44]  E. Stern,et al.  Linguistic threat activates the human amygdala. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[45]  Penny M. Pexman,et al.  Evidence for the activation of sensorimotor information during visual word recognition: The body–object interaction effect , 2008, Cognition.

[46]  Lance J. Rips,et al.  Structure and process in semantic memory: A featural model for semantic decisions. , 1974 .

[47]  Michael P. Kaschak,et al.  Putting words in perspective , 2004, Memory & cognition.

[48]  Rolf A. Zwaan,et al.  Spatial iconicity affects semantic relatedness judgments , 2003, Psychonomic bulletin & review.

[49]  Rolf A. Zwaan,et al.  PSYCHOLOGICAL SCIENCE Research Article THE EFFECT OF IMPLIED ORIENTATION DERIVED FROM VERBAL CONTEXT ON PICTURE RECOGNITION , 2022 .

[50]  J. Hodges,et al.  Apraxia, mechanical problem solving and semantic knowledge , 2002, Journal of Neurology.

[51]  Srini Narayanan,et al.  Spatial and Linguistic Aspects of Visual Imagery in Sentence Comprehension , 2007, Cogn. Sci..

[52]  M. Ross Quillian,et al.  Retrieval time from semantic memory , 1969 .

[53]  Bradford Z. Mahon,et al.  A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content , 2008, Journal of Physiology-Paris.

[54]  Leslie G. Ungerleider,et al.  Discrete Cortical Regions Associated with Knowledge of Color and Knowledge of Action , 1995, Science.

[55]  G. Lakoff,et al.  The Brain's concepts: the role of the Sensory-motor system in conceptual knowledge , 2005, Cognitive neuropsychology.

[56]  F. Pulvermüller,et al.  Category-specific conceptual processing of color and form in left fronto-temporal cortex. , 2006, Cerebral cortex.

[57]  R L Klatzky,et al.  Rubbing your stomach while tapping your fingers: interference between motor planning and semantic judgments. , 1992, Journal of experimental psychology. Human perception and performance.

[58]  William S Horton,et al.  Out of sight, out of mind: Occlusion and the accessibility of information in narrative comprehension , 2003, Psychonomic bulletin & review.

[59]  James L. McClelland,et al.  A computational model of semantic memory impairment: modality specificity and emergent category specificity. , 1991 .

[60]  S. Scott,et al.  The role of semantics and grammatical class in the neural representation of words. , 2006, Cerebral cortex.

[61]  A. Roepstorff,et al.  Motion verb sentences activate left posterior middle temporal cortex despite static context , 2005, Neuroreport.

[62]  M. Garrett,et al.  Representing the meanings of object and action words: The featural and unitary semantic space hypothesis , 2004, Cognitive Psychology.

[63]  F. Pulvermüller,et al.  Words in the brain's language , 1999, Behavioral and Brain Sciences.

[64]  Friedemann Pulvermüller,et al.  Clinical, imaging and pathological correlates of a hereditary deficit in verb and action processing. , 2006, Brain : a journal of neurology.

[65]  T. Shallice,et al.  Category specific semantic impairments , 1984 .

[66]  Mark S. Seidenberg,et al.  On the nature and scope of featural representations of word meaning. , 1997, Journal of experimental psychology. General.

[67]  J. Haxby,et al.  Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects , 1999, Nature Neuroscience.

[68]  Allen Newell,et al.  Physical Symbol Systems , 1980, Cogn. Sci..

[69]  A. Damasio,et al.  Neural Correlates of Naming Actions and of Naming Spatial Relations , 2001, NeuroImage.

[70]  J. Raaijmakers,et al.  Does pizza prime coin? Perceptual priming in lexical decision and pronunciation. , 1998 .

[71]  Bahador Bahrami,et al.  Visual motion interferes with lexical decision on motion words , 2008, Current Biology.

[72]  Yuka Sasaki,et al.  Greater Disruption Due to Failure of Inhibitory Control on an Ambiguous Distractor , 2022 .

[73]  M. Tanenhaus,et al.  Actions and affordances in syntactic ambiguity resolution. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[74]  Rolf A. Zwaan,et al.  Processing negated sentences with contradictory predicates: Is a door that is not open mentally closed? , 2006 .

[75]  G. Bower,et al.  Temporal and spatial distance in situation models , 2000, Memory & cognition.

[76]  Michael P. Kaschak,et al.  Perception of motion affects language processing , 2005, Cognition.

[77]  G. Rizzolatti,et al.  Listening to action-related sentences modulates the activity of the motor system: a combined TMS and behavioral study. , 2005, Brain research. Cognitive brain research.

[78]  Rolf A. Zwaan,et al.  Moving words: dynamic representations in language comprehension , 2004 .

[79]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[80]  G. Rizzolatti,et al.  Congruent Embodied Representations for Visually Presented Actions and Linguistic Phrases Describing Actions , 2006, Current Biology.

[81]  Teenie Matlock,et al.  Do eye movements go with fictive motion , 2004 .

[82]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[83]  L. Barsalou,et al.  Perceptual simulation in property verification , 2004, Memory & cognition.

[84]  Allan Collins,et al.  A spreading-activation theory of semantic processing , 1975 .

[85]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[86]  L. Barsalou,et al.  Head Up, Foot Down , 2008, Psychological science.

[87]  L. Connell Representing object colour in language comprehension , 2007, Cognition.

[88]  G S Dell,et al.  A spreading-activation theory of retrieval in sentence production. , 1986, Psychological review.

[89]  F. Pulvermüller,et al.  Walking or Talking?: Behavioral and Neurophysiological Correlates of Action Verb Processing , 2001, Brain and Language.

[90]  B. Laeng,et al.  Eye scanpaths during visual imagery reenact those of perception of the same visual scene , 2002 .

[91]  Daniel C. Richardson,et al.  Spatial representations activated during real-time comprehension of verbs , 2003, Cogn. Sci..

[92]  T. Talavage,et al.  Neuroanatomical distribution of five semantic components of verbs: Evidence from fMRI , 2008, Brain and Language.

[93]  Alfonso Caramazza,et al.  The multiple semantics hypothesis: Multiple confusions? , 1990 .

[94]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[95]  Irene P. Kan,et al.  ROLE OF MENTAL IMAGERY IN A PROPERTY VERIFICATION TASK: FMRI EVIDENCE FOR PERCEPTUAL REPRESENTATIONS OF CONCEPTUAL KNOWLEDGE , 2003, Cognitive neuropsychology.

[96]  G. Bower,et al.  The metrics of spatial situation models. , 1997, Journal of experimental psychology. Learning, memory, and cognition.

[97]  Michael P. Kaschak,et al.  The Body's Contribution to Language , 2003 .

[98]  Bradford Z. Mahon,et al.  The organization of conceptual knowledge: the evidence from category-specific semantic deficits , 2003, Trends in Cognitive Sciences.

[99]  Alfonso Caramazza,et al.  The orchestration of the sensory-motor systems: Clues from Neuropsychology , 2005, Cognitive neuropsychology.

[100]  Alex Martin,et al.  Semantic memory and the brain: structure and processes , 2001, Current Opinion in Neurobiology.

[101]  James L. McClelland,et al.  Structure and deterioration of semantic memory: a neuropsychological and computational investigation. , 2004, Psychological review.

[102]  Alex Martin,et al.  Representation of Manipulable Man-Made Objects in the Dorsal Stream , 2000, NeuroImage.

[103]  Rolf A. Zwaan,et al.  Embodied Language: A Review of the Role of the Motor System in Language Comprehension , 2008, Quarterly journal of experimental psychology.

[104]  Zenon W. Pylyshyn,et al.  Computation and Cognition: Toward a Foundation for Cognitive Science , 1984 .

[105]  Diane Pecher,et al.  Sensorimotor simulations underlie conceptual representations: Modality-specific effects of prior activation , 2004, Psychonomic bulletin & review.

[106]  J. Fodor Psychosemantics: The Problem of Meaning in the Philosophy of Mind , 1988 .

[107]  R. Henson,et al.  Multiple levels of visual object constancy revealed by event-related fMRI of repetition priming , 2002, Nature Neuroscience.

[108]  Rebecca Fincher-Kiefer,et al.  Perceptual components of situation models , 2001, Memory & cognition.

[109]  R. Ellis,et al.  On the relations between seen objects and components of potential actions. , 1998, Journal of experimental psychology. Human perception and performance.

[110]  Rolf A. Zwaan,et al.  Seeing, acting, understanding: motor resonance in language comprehension. , 2006, Journal of experimental psychology. General.

[111]  Rolf A. Zwaan Situation Models , 1999 .

[112]  Leslie G. Ungerleider,et al.  Neural correlates of category-specific knowledge , 1996, Nature.

[113]  A. Damasio Category-related recognition defects as a clue to the neural substrates of knowledge , 1990, Trends in Neurosciences.

[114]  Marc Jeannerod,et al.  Cross-talk between Language Processes and Overt Motor Behavior in the First 200 msec of Processing , 2006, Journal of Cognitive Neuroscience.

[115]  E. Warrington,et al.  Categories of knowledge. Further fractionations and an attempted integration. , 1987, Brain : a journal of neurology.