The activation of modality-specific representations during discourse processing

Previous research has shown that readers generate mental images of events. Most studies have investigated imagery during the reading of short texts, which also included explicit judgment tasks. In two fMRI studies, we assessed whether modality-specific imagery occurs during naturalistic, discourse comprehension. We identified clauses in the texts that elicited auditory, motor, or visual imagery. In both studies, reading motor imagery clauses was associated with increases in activity in left postcentral and precentral sulci, and reading auditory imagery clauses was associated with increases in left superior temporal gyrus and perisylvian language-related regions. Study 2 compared presentation of connected discourse to a condition in which unconnected sentences were presented, preventing the establishment of global coherence. Sensorimotor imagery was strongest when readers were able to generate a globally coherent discourse representation. Overall, these results suggest that modality-specific imagery occurs during discourse comprehension and it is dependent on the development of discourse-level representations.

[1]  Josh H. McDermott,et al.  Functional imaging of human visual recognition. , 1996, Brain research. Cognitive brain research.

[2]  U. Castiello.  The neuroscience of grasping , 2005, Nature Reviews Neuroscience.

[3]  Timothy L Hubbard,et al.  Auditory imagery: empirical findings. , 2010, Psychological bulletin.

[4]  M. Diamond,et al.  Primary Motor and Sensory Cortex Activation during Motor Performance and Motor Imagery: A Functional Magnetic Resonance Imaging Study , 1996, The Journal of Neuroscience.

[5]  Vittorio Gallese,et al.  Listening to Action-related Sentences Activates Fronto-parietal Motor Circuits , 2005, Journal of Cognitive Neuroscience.

[6]  G. Hesslow.  Conscious thought as simulation of behaviour and perception , 2002, Trends in Cognitive Sciences.

[7]  M. Brett,et al.  Large, colorful, or noisy? Attribute- and modality-specific activations during retrieval of perceptual attribute knowledge , 2001 .

[8]  Rolf A. Zwaan,et al.  Language Comprehenders Mentally Represent the Shapes of Objects , 2002, Psychological science.

[9]  R. Zatorre,et al.  Behavioral and neural correlates of perceived and imagined musical timbre , 2004, Neuropsychologia.

[10]  M J Brammer,et al.  A functional study of auditory verbal imagery , 2001, Psychological Medicine.

[11]  Alan C. Evans,et al.  Hearing in the Mind's Ear: A PET Investigation of Musical Imagery and Perception , 1996, Journal of Cognitive Neuroscience.

[12]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[13]  D. Stuss,et al.  The frontal lobes are necessary for 'theory of mind'. , 2001, Brain : a journal of neurology.

[14]  Russell A Poldrack,et al.  Functional MRI at the crossroads. , 2009, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[15]  T. Allison,et al.  Social perception from visual cues: role of the STS region , 2000, Trends in Cognitive Sciences.

[16]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[17]  David Silbersweig,et al.  Functional anatomy of inner speech and auditory verbal imagery , 1995, Schizophrenia Research.

[18]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[19]  Z. Pylyshyn.  The imagery debate: Analogue media versus tacit knowledge. , 1981 .

[20]  Daniel Casasanto,et al.  Neural Dissociations between Action Verb Understanding and Motor Imagery , 2010, Journal of Cognitive Neuroscience.

[21]  Jeffrey M. Zacks,et al.  Human Brain Activity Time-Locked to Narrative Event Boundaries , 2007, Psychological science.

[22]  L. Barsalou,et al.  Verifying Different-Modality Properties for Concepts Produces Switching Costs , 2003, Psychological science.

[23]  Bo Yao,et al.  Silent Reading of Direct versus Indirect Speech Activates Voice-selective Areas in the Auditory Cortex , 2011, Journal of Cognitive Neuroscience.

[24]  Jason E. Albrecht,et al.  Updating a mental model: maintaining both local and global coherence , 1993 .

[25]  M. Farah,et al.  A functional MRI study of mental image generation , 1997, Neuropsychologia.

[26]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[27]  Mark S. Seidenberg,et al.  Activation of sensory-motor areas in sentence comprehension. , 2010, Cerebral cortex.

[28]  Rolf A. Zwaan,et al.  Seeing, acting, understanding: motor resonance in language comprehension. , 2006, Journal of experimental psychology. General.

[29]  R. Buckner,et al.  Cluster size thresholds for assessment of significant activation in fMRI , 2001, NeuroImage.

[30]  Franz Schmalhofer,et al.  Neural Indicators of Inference Processes in Text Comprehension: An Event-related Functional Magnetic Resonance Imaging Study , 2008, Journal of Cognitive Neuroscience.

[31]  R. Knight,et al.  Frontal Lobe Contributions to Theory of Mind , 1998, Journal of Cognitive Neuroscience.

[32]  Joseph P. Magliano,et al.  Those voices in your head: Activation of auditory images during reading , 2009, Cognition.

[33]  W. K. Simmons,et al.  A common neural substrate for perceiving and knowing about color , 2007, Neuropsychologia.

[34]  Jeffrey M. Zacks,et al.  Lateral somatotopic organization during imagined and prepared movements. , 2006, Journal of neurophysiology.

[35]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[36]  Rolf A. Zwaan,et al.  Embodied Language: A Review of the Role of the Motor System in Language Comprehension , 2008, Quarterly journal of experimental psychology.

[37]  G. Rizzolatti,et al.  Congruent Embodied Representations for Visually Presented Actions and Linguistic Phrases Describing Actions , 2006, Current Biology.

[38]  M. Kiefer,et al.  The Sound of Concepts: Four Markers for a Link between Auditory and Conceptual Brain Systems , 2008, The Journal of Neuroscience.

[39]  P. Johnson-Laird,et al.  Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness , 1985 .

[40]  T. Trabasso,et al.  Constructing inferences during narrative text comprehension. , 1994, Psychological review.

[41]  Derek K. Jones,et al.  Perisylvian language networks of the human brain , 2005, Annals of neurology.

[42]  Colin M. Brown,et al.  Semantic Integration in Sentences and Discourse: Evidence from the N400 , 1999, Journal of Cognitive Neuroscience.

[43]  Rolf A. Zwaan.  The Immersed Experiencer: Toward An Embodied Theory Of Language Comprehension , 2003 .

[44]  Evelyn C. Ferstl,et al.  The extended language network: A meta‐analysis of neuroimaging studies on text comprehension , 2008, Human brain mapping.

[45]  D. V. van Essen,et al.  A Population-Average, Landmark- and Surface-based (PALS) atlas of human cerebral cortex. , 2005, NeuroImage.

[46]  M. Louwerse,et al.  The linguistic and embodied nature of conceptual processing , 2010, Cognition.

[47]  A. Glenberg,et al.  What memory is for: Creating meaning in the service of action , 1997, Behavioral and Brain Sciences.

[48]  David F. Marks,et al.  New directions for mental imagery research. , 1995 .

[49]  Seung-Schik Yoo,et al.  Human brain mapping of auditory imagery: event-related functional MRI study , 2001, Neuroreport.

[50]  Rolf A. Zwaan,et al.  PSYCHOLOGICAL SCIENCE Research Article THE EFFECT OF IMPLIED ORIENTATION DERIVED FROM VERBAL CONTEXT ON PICTURE RECOGNITION , 2022 .

[51]  Michael P. Kaschak,et al.  Perception of motion affects language processing , 2005, Cognition.

[52]  David C. Van Essen,et al.  A Population-Average, Landmark- and Surface-based (PALS) atlas of human cerebral cortex , 2005, NeuroImage.

[53]  Bradford Z. Mahon,et al.  A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content , 2008, Journal of Physiology-Paris.

[54]  Lawrence W. Barsalou,et al.  Language and simulation in conceptual processing , 2008 .

[55]  R. Poldrack.  Can cognitive processes be inferred from neuroimaging data? , 2006, Trends in Cognitive Sciences.

[56]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[57]  Tad T. Brunyé,et al.  You heard it here first: readers mentally simulate described sounds. , 2010, Acta psychologica.

[58]  J. Decety,et al.  Functional anatomy of execution, mental simulation, observation, and verb generation of actions: A meta‐analysis , 2001, Human brain mapping.

[59]  Jeffrey M. Zacks,et al.  Human brain activity time-locked to perceptual event boundaries , 2001, Nature Neuroscience.

[60]  M. Louwerse.  Embodied relations are encoded in language , 2008, Psychonomic bulletin & review.

[61]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[62]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[63]  M. Farah,et al.  A neural basis for category and modality specificity of semantic knowledge , 1999, Neuropsychologia.

[64]  Katie L. McMahon,et al.  Action word meaning representations in cytoarchitectonically defined primary and premotor cortices , 2008, NeuroImage.

[65]  Ludovica Labruna,et al.  A Functional Role for the Motor System in Language Understanding , 2011, Psychological science.

[66]  Uta Frith,et al.  Theory of mind , 2001, Current Biology.

[67]  Rolf A. Zwaan,et al.  Moving words: dynamic representations in language comprehension , 2004 .

[68]  Jeffrey M. Zacks,et al.  Neural substrates of narrative comprehension and memory , 2008, NeuroImage.

[69]  A. Glenberg,et al.  Action-based language: A theory of language acquisition, comprehension, and production , 2012, Cortex.

[70]  Rolf A. Zwaan,et al.  Spatial iconicity affects semantic relatedness judgments , 2003, Psychonomic bulletin & review.

[71]  A. Caramazza,et al.  Concepts Are More than Percepts: The Case of Action Verbs , 2008, The Journal of Neuroscience.

[72]  J. Talairach,et al.  Co-Planar Stereotaxic Atlas of the Human Brain: 3-Dimensional Proportional System: An Approach to Cerebral Imaging , 1988 .

[73]  D. Kemmerer,et al.  The Two-Level Theory of verb meaning: An approach to integrating the semantics of action with the mirror neuron system , 2010, Brain and Language.

[74]  A. Damasio,et al.  Neural Correlates of Naming Actions and of Naming Spatial Relations , 2001, NeuroImage.

[75]  R. Barker,et al.  One boy's day : a specimen record of behavior , 1951 .

[76]  Rebecca Fincher-Kiefer,et al.  Perceptual components of situation models , 2001, Memory & cognition.

[77]  Rolf A. Zwaan,et al.  Discourse comprehension. , 1997, Annual review of psychology.

[78]  S. Petersen,et al.  Memory's echo: vivid remembering reactivates sensory-specific cortex. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[79]  Jeffrey M. Zacks,et al.  Reading Stories Activates Neural Representations of Visual and Motor Experiences , 2009, Psychological science.

[80]  R. Blake,et al.  Brain Areas Involved in Perception of Biological Motion , 2000, Journal of Cognitive Neuroscience.

[81]  G. Orban,et al.  Comparative mapping of higher visual areas in monkeys and humans , 2004, Trends in Cognitive Sciences.

[82]  R. Zatorre,et al.  When that tune runs through your head: a PET investigation of auditory imagery for familiar melodies. , 1999, Cerebral cortex.

[83]  David C. Van Essen,et al.  Application of Information Technology: An Integrated Software Suite for Surface-based Analyses of Cerebral Cortex , 2001, J. Am. Medical Informatics Assoc..

[84]  R Saxe,et al.  People thinking about thinking people The role of the temporo-parietal junction in “theory of mind” , 2003, NeuroImage.

[85]  Stefan Geyer,et al.  Imagery of voluntary movement of fingers, toes, and tongue activates corresponding body-part-specific motor representations. , 2003, Journal of neurophysiology.

[86]  G. Lakoff,et al.  The Brain's concepts: the role of the Sensory-motor system in conceptual knowledge , 2005, Cognitive neuropsychology.

[87]  Anjan Chatterjee,et al.  Specificity of Action Representations in the Lateral Occipitotemporal Cortex , 2006, Journal of Cognitive Neuroscience.

[88]  A. Markman,et al.  Constraining Theories of Embodied Cognition , 2005, Psychological science.

[89]  D. Heeger,et al.  Linear Systems Analysis of Functional Magnetic Resonance Imaging in Human V1 , 1996, The Journal of Neuroscience.

[90]  Z. Pylyshyn.  Mental imagery: In search of a theory , 2002, Behavioral and Brain Sciences.

[91]  J. Kable,et al.  Neural Substrates of Action Event Knowledge , 2002, Journal of Cognitive Neuroscience.

[92]  Roel M. Willems,et al.  Body-specific representations of action verbs: Neural evidence from right- and left-handers , 2009, NeuroImage.

[93]  D. McDermott.  LANGUAGE OF THOUGHT , 2012 .

[94]  Keith Oatley,et al.  The Function of Fiction is the Abstraction and Simulation of Social Experience , 2008, Perspectives on psychological science : a journal of the Association for Psychological Science.