Modulation of BOLD Response in Motion-sensitive Lateral Temporal Cortex by Real and Fictive Motion Sentences

Can linguistic semantics affect neural processing in feature-specific visual regions? Specifically, when we hear a sentence describing a situation that includes motion, do we engage neural processes that are part of the visual perception of motion? How about if a motion verb was used figuratively, not literally? We used fMRI to investigate whether semantic content can “penetrate” and modulate neural populations that are selective to specific visual properties during natural language comprehension. Participants were presented audiovisually with three kinds of sentences: motion sentences (“The wild horse crossed the barren field.”), static sentences, (“The black horse stood in the barren field.”), and fictive motion sentences (“The hiking trail crossed the barren field.”). Motion-sensitive visual areas (MT+) were localized individually in each participant as well as face-selective visual regions (fusiform face area; FFA). MT+ was activated significantly more for motion sentences than the other sentence types. Fictive motion sentences also activated MT+ more than the static sentences. Importantly, no modulation of neural responses was found in FFA. Our findings suggest that the neural substrates of linguistic semantics include early visual areas specifically related to the represented semantics and that figurative uses of motion verbs also engage these neural systems, but to a lesser extent. These data are consistent with a view of language comprehension as an embodied process, with neural substrates as far reaching as early sensory brain areas that are specifically related to the represented semantics.

[1]  Ravi S. Menon,et al.  Distinguishing subregions of the human MT+ complex using visual fields and pursuit eye movements. , 2001, Journal of neurophysiology.

[2]  L. Tyler,et al.  Towards a distributed account of conceptual knowledge , 2001, Trends in Cognitive Sciences.

[3]  J. Winawer,et al.  A Motion Aftereffect From Still Photographs Depicting Motion , 2008, Psychological science.

[4]  E. Bullmore,et al.  The functional neuroanatomy of implicit-motion perception or ‘representational momentum’ , 2000, Current Biology.

[5]  E. G. Jones Cerebral Cortex , 1987, Cerebral Cortex.

[6]  Lera Boroditsky,et al.  Visual motion aftereffect from understanding motion language , 2010, Proceedings of the National Academy of Sciences.

[7]  T. Matlock Fictive motion as cognitive simulation , 2004, Memory & cognition.

[8]  Teenie Matlock,et al.  The integration of figurative language and static depictions: An eye movement study of fictive motion , 2007, Cognition.

[9]  G. Lakoff,et al.  Metaphors We Live by , 1981 .

[10]  G. Orban,et al.  Charting the Lower Superior Temporal Region, a New Motion-Sensitive Region in Monkey Superior Temporal Sulcus , 2006, The Journal of Neuroscience.

[11]  Leslie G. Ungerleider,et al.  Cortical connections of visual area MT in the macaque , 1986, The Journal of comparative neurology.

[12]  B. Avants,et al.  Impaired action knowledge in amyotrophic lateral sclerosis , 2008, Neurology.

[13]  Alfonso Caramazza,et al.  Perception, action, and word meanings in the human brain: the case from action verbs , 2011, Annals of the New York Academy of Sciences.

[14]  B. Bahrami,et al.  Motion Detection and Motion Verbs , 2007, Psychological science.

[15]  E. DeYoe,et al.  A comparison of visual and auditory motion processing in human cerebral cortex. , 2000, Cerebral cortex.

[16]  R. Passingham,et al.  Seeing or Doing? Influence of Visual and Motor Familiarity in Action Observation , 2006, Current Biology.

[17]  S. Albert,et al.  Nouns, verbs, and ALS , 2008, Neurology.

[18]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[19]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[20]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[21]  A. Glenberg,et al.  Processing Abstract Language Modulates Motor System Activity , 2008, Quarterly journal of experimental psychology.

[22]  A. Glenberg,et al.  Action-based language: A theory of language acquisition, comprehension, and production , 2012, Cortex.

[23]  Sharon L. Thompson-Schill,et al.  Conceptual Representations of Action in the Lateral Temporal Cortex , 2005, Journal of Cognitive Neuroscience.

[24]  M. Sereno,et al.  From monkeys to humans: what do we now know about brain homologies? , 2005, Current Opinion in Neurobiology.

[25]  N. Kanwisher,et al.  Activation in Human MT/MST by Static Images with Implied Motion , 2000, Journal of Cognitive Neuroscience.

[26]  Michael K. Tanenhaus,et al.  Neural correlates of partial lexical activation , 2008, Proceedings of the National Academy of Sciences.

[27]  Frank Bremmer,et al.  Neural correlates of implied motion , 2003, Nature.

[28]  Page Widick,et al.  Functional–anatomical organization of predicate metaphor processing , 2008, Brain and Language.

[29]  Bradford Z. Mahon,et al.  A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content , 2008, Journal of Physiology-Paris.

[30]  Carol A. Seger,et al.  Right hemisphere metaphor processing? Characterizing the lateralization of semantic processes , 2007, Brain and Language.

[31]  Leslie G. Ungerleider,et al.  Discrete Cortical Regions Associated with Knowledge of Color and Knowledge of Action , 1995, Science.

[32]  M. Talarowska,et al.  [The role of the right hemisphere in the aetiology of the depressive disorders]. , 2011, Psychiatria polska.

[33]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[34]  Friedemann Pulvermüller,et al.  Reading cinnamon activates olfactory brain regions , 2006, NeuroImage.

[35]  J. Fodor,et al.  The Modularity of Mind: An Essay on Faculty Psychology , 1984 .

[36]  L. Boroditsky Metaphoric structuring: understanding time through spatial metaphors , 2000, Cognition.

[37]  Rolf A. Zwaan,et al.  Language Comprehenders Mentally Represent the Shapes of Objects , 2002, Psychological science.

[38]  W. K. Simmons,et al.  A common neural substrate for perceiving and knowing about color , 2007, Neuropsychologia.

[39]  Anjan Chatterjee,et al.  From novel to familiar: Tuning the brain for metaphors , 2012, NeuroImage.

[40]  M. Jeannerod Neural Simulation of Action: A Unifying Mechanism for Motor Cognition , 2001, NeuroImage.

[41]  Steven L. Small,et al.  Interpretation-mediated changes in neural activity during language comprehension , 2011, NeuroImage.

[42]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[43]  A. Damasio Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition , 1989, Cognition.

[44]  Bahador Bahrami,et al.  Visual motion interferes with lexical decision on motion words , 2008, Current Biology.

[45]  J. Fodor The Modularity of mind. An essay on faculty psychology , 1986 .

[46]  Erik Asp,et al.  Naming dynamic and static actions: Neuropsychological evidence , 2008, Journal of Physiology-Paris.

[47]  A. Caramazza,et al.  Domain-Specific Knowledge Systems in the Brain: The Animate-Inanimate Distinction , 1998, Journal of Cognitive Neuroscience.

[48]  P. Reber,et al.  Correction of off resonance‐related distortion in echo‐planar imaging using EPI‐based field maps , 1998, Magnetic resonance in medicine.

[49]  Marina Pavlova,et al.  Neuromagnetic Response to Body Motion and Brain Connectivity , 2009, Journal of Cognitive Neuroscience.

[50]  Daniel C. Richardson,et al.  Spatial representations activated during real-time comprehension of verbs , 2003, Cogn. Sci..

[51]  T. Talavage,et al.  Neuroanatomical distribution of five semantic components of verbs: Evidence from fMRI , 2008, Brain and Language.

[52]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[53]  A. Caramazza,et al.  Concepts Are More than Percepts: The Case of Action Verbs , 2008, The Journal of Neuroscience.

[54]  C. Miniussi,et al.  Literal, fictive and metaphorical motion sentences preserve the motion component of the verb: A TMS study , 2011, Brain and Language.

[55]  K. Zentgraf,et al.  Cognitive motor processes: The role of motor imagery in the study of motor representations , 2009, Brain Research Reviews.

[56]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[57]  Karl J. Friston,et al.  Action selectivity in parietal and temporal cortex. , 2005, Brain research. Cognitive brain research.

[58]  Richard S. J. Frackowiak,et al.  Area V5 of the human brain: evidence from a combined study using positron emission tomography and magnetic resonance imaging. , 1993, Cerebral cortex.

[59]  A. Damasio,et al.  Neural Correlates of Naming Actions and of Naming Spatial Relations , 2001, NeuroImage.

[60]  D. Heeger,et al.  Retinotopy and Functional Subdivision of Human Areas MT and MST , 2002, The Journal of Neuroscience.

[61]  Billi Randall,et al.  Cortical Differentiation for Nouns and Verbs Depends on Grammatical Markers , 2008, Journal of Cognitive Neuroscience.

[62]  Antonia F. de C. Hamilton,et al.  Action Understanding Requires the Left Inferior Frontal Cortex , 2006, Current Biology.

[63]  B. Mesquita,et al.  Adjustment to Chronic Diseases and Terminal Illness Health Psychology : Psychological Adjustment to Chronic Disease , 2006 .

[64]  Elizabeth Bates,et al.  Action comprehension in aphasia: linguistic and non-linguistic deficits and their lesion correlates , 2004, Neuropsychologia.

[65]  A. Schleicher,et al.  Cytoarchitectonic analysis of the human extrastriate cortex in the region of V5/MT+: a probabilistic, stereotaxic map of area hOc5. , 2006, Cerebral cortex.

[66]  Michael P. Kaschak,et al.  Perception of motion affects language processing , 2005, Cognition.

[67]  Angela D. Friederici,et al.  Comprehending Prehending: Neural Correlates of Processing Verbs with Motor Stems , 2007, Journal of Cognitive Neuroscience.

[68]  R. Adolphs,et al.  NEURAL CORRELATES OF CONCEPTUAL KNOWLEDGE FOR ACTIONS , 2003, Cognitive neuropsychology.

[69]  Matthew H. Davis,et al.  Is there an anatomical basis for category-specificity? Semantic memory studies in PET and fMRI , 2002, Neuropsychologia.

[70]  Rosaleen A. McCarthy,et al.  Multiple meaning systems in the brain: A case for visual semantics , 1994, Neuropsychologia.

[71]  S. Kosslyn,et al.  Visual mental imagery induces retinotopically organized activation of early visual areas. , 2005, Cerebral cortex.

[72]  M. Jeannerod,et al.  Word processing in Parkinson's disease is impaired for action verbs but not for concrete nouns , 2008, Neuropsychologia.

[73]  W. K. Simmons,et al.  Pictures of appetizing foods activate gustatory cortices for taste and reward. , 2005, Cerebral cortex.

[74]  J. Moake,et al.  This article has been cited by other articles , 2003 .

[75]  M. Jung-Beeman Bilateral brain processes for comprehending natural language , 2005, Trends in Cognitive Sciences.

[76]  F. Pulvermüller,et al.  Grasping Ideas with the Motor System: Semantic Somatotopy in Idiom Comprehension , 2009 .

[77]  R. Andersen,et al.  Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[78]  J. Kable,et al.  Neural Substrates of Action Event Knowledge , 2002, Journal of Cognitive Neuroscience.

[79]  Alan C. Evans,et al.  A new anatomical landmark for reliable identification of human area V5/MT: a quantitative analysis of sulcal patterning. , 2000, Cerebral cortex.

[80]  P. McGuire,et al.  Neural systems underlying British Sign Language and audio-visual English processing in native users. , 2002, Brain : a journal of neurology.

[81]  Srini Narayanan,et al.  Spatial and Linguistic Aspects of Visual Imagery in Sentence Comprehension , 2007, Cogn. Sci..

[82]  Friedemann Pulvermüller,et al.  Brain mechanisms linking language and action , 2005, Nature Reviews Neuroscience.

[83]  Leonard Talmy,et al.  Toward a cognitive semantics, Vol. 1: Concept structuring systems. , 2000 .

[84]  A. Roepstorff,et al.  Motion verb sentences activate left posterior middle temporal cortex despite static context , 2005, Neuroreport.

[85]  Karl J. Friston,et al.  A direct demonstration of functional specialization in human visual cortex , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[86]  C. Petten,et al.  A special role for the right hemisphere in metaphor comprehension? ERP evidence from hemifield presentation , 2007, Brain Research.

[87]  Jeffrey R. Binder,et al.  The Neural Career of Sensory-motor Metaphors , 2011, Journal of Cognitive Neuroscience.

[88]  T. Hendler,et al.  The role of the right hemisphere in processing nonsalient metaphorical meanings: Application of Principal Components Analysis to fMRI data , 2005, Neuropsychologia.

[89]  Richard S. J. Frackowiak,et al.  The role of the right hemisphere in the interpretation of figurative aspects of language. A positron emission tomography activation study. , 1994, Brain : a journal of neurology.

[90]  M. Giese,et al.  Nonvisual Motor Training Influences Biological Motion Perception , 2006, Current Biology.

[91]  Lisa Aziz-Zadeh,et al.  Embodied semantics for actions: Findings from functional brain imaging , 2008, Journal of Physiology-Paris.

[92]  A. Saygin Superior temporal and premotor brain areas necessary for biological motion perception. , 2007, Brain : a journal of neurology.

[93]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[94]  G. Rizzolatti,et al.  Congruent Embodied Representations for Visually Presented Actions and Linguistic Phrases Describing Actions , 2006, Current Biology.

[95]  Friedemann Pulvermüller,et al.  Motor cortex maps articulatory features of speech sounds , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[96]  Rolf A. Zwaan,et al.  Seeing, acting, understanding: motor resonance in language comprehension. , 2006, Journal of experimental psychology. General.

[97]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[98]  Craig J. Brozinsky,et al.  Impact of Early Deafness and Early Exposure to Sign Language on the Cerebral Organization for Motion Processing , 2001, The Journal of Neuroscience.

[99]  Alfonso Caramazza,et al.  NEUROPSYCHOLOGICAL AND NEUROIMAGING PERSPECTIVES ON CONCEPTUAL KNOWLEDGE: AN INTRODUCTION , 2003, Cognitive neuropsychology.

[100]  H. Damasio,et al.  Effects of noun–verb homonymy on the neural correlates of naming concrete entities and actions , 2005, Brain and Language.

[101]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[102]  Michael S Beauchamp,et al.  See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex , 2005, Current Opinion in Neurobiology.

[103]  S. Kosslyn,et al.  When is early visual cortex activated during visual mental imagery? , 2003, Psychological bulletin.

[104]  Jennifer E. Aldrich,et al.  See Me! Hear Me! Touch Me! Move Me! , 2006 .

[105]  A. Caramazza,et al.  The organisation of conceptual knowledge in the brain: The future's past and some future directions , 2006, Cognitive neuropsychology.

[106]  Vittorio Gallese,et al.  Listening to Action-related Sentences Activates Fronto-parietal Motor Circuits , 2005, Journal of Cognitive Neuroscience.

[107]  M. Sereno,et al.  Point-Light Biological Motion Perception Activates Human Premotor Cortex , 2004, The Journal of Neuroscience.

[108]  L. Tyler,et al.  Modulation of motor and premotor cortices by actions, action words and action sentences , 2009, Neuropsychologia.

[109]  J. Haxby,et al.  Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects , 1999, Nature Neuroscience.

[110]  Mirella Dapretto,et al.  Metaphorical vs. literal word meanings: fMRI evidence against a selective role of the right hemisphere , 2006, NeuroImage.