Picturing meaning: an ERP study on the integration of left or right-handed first-person perspective pictures into a sentence context

ABSTRACT Verbal and pictorial information are often processed together. Therefore, knowing how and when information from these modalities is integrated is important. In this ERP study we investigated integration of pictorial information into a sentence context. Right-handed participants heard sentences containing manual action verbs (e.g. “You are slicing the tomato”), while seeing a picture of a manual action. Pictures matched or mismatched the sentence content and the participants’ handedness (i.e. pictures showed a left or right-handed perspective). Results showed a larger N400-amplitude for content-mismatching than for content-matching sentence-picture pairs. The N400-amplitude was not larger when the picture mismatched the participants’ handedness. However, participants responded faster to right than to left-handed perspective pictures. This study suggests that with a sentence context, pictures are integrated with verbal information, but mental simulations either do not play a role in this process or this role might be too small to be visualised in the N400.

[1]  Rolf A. Zwaan.  Embodiment and language comprehension: reframing the discussion , 2014, Trends in Cognitive Sciences.

[2]  Fred Paas,et al.  When Left Is Not Right , 2013, Psychological science.

[3]  B. Bergen,et al.  The case of the missing pronouns: Does mentally simulated perspective play a functional role in the comprehension of person? , 2013, Cognition.

[4]  F. Manes,et al.  N400 ERPs for actions: building meaning in context , 2013, Front. Hum. Neurosci..

[5]  Angelo Cangelosi,et al.  Object affordance influences instruction span , 2012, Experimental Brain Research.

[6]  Marta Kutas,et al.  Comprehending how visual context influences incremental sentence processing: Insights from ERPs and picture-sentence verification. , 2011, Psychophysiology.

[7]  Alan Garnham,et al.  Switching Modalities in A Sentence Verification Task: ERP Evidence for Embodied Language Processing , 2011, Front. Psychology.

[8]  L. Deouell,et al.  ERP evidence for context congruity effects during simultaneous object–scene processing , 2010, Neuropsychologia.

[9]  Jeanine K. Stefanucci,et al.  The effects of handedness and reachability on perceived distance. , 2009, Journal of experimental psychology. Human perception and performance.

[10]  Alice Mado Proverbio,et al.  RP and N400 ERP components reflect semantic violations in visual processing of human actions , 2009, Neuroscience Letters.

[11]  D. Casasanto,et al.  of Experimental Psychology , 2022 .

[12]  Rick Dale,et al.  Explanatory Pluralism in Cognitive Science , 2009, Cogn. Sci..

[13]  W. Prinz,et al.  N400-like negativities in action perception reflect the activation of two components of an action representation , 2009, Social neuroscience.

[14]  Jiro Gyoba,et al.  Event-related potentials during the evaluation of the appropriateness of cooperative actions , 2009, Neuroscience Letters.

[15]  Guy Dove.  Beyond perceptual symbols: A call for representational pluralism , 2009, Cognition.

[16]  Roel M. Willems,et al.  Seeing and Hearing Meaning: ERP and fMRI Evidence of Word versus Picture Integration into a Sentence Context , 2008, Journal of Cognitive Neuroscience.

[17]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[18]  P. Hagoort,et al.  Thematic role assignment in patients with Broca's aphasia: Sentence–picture matching electrified , 2007, Neuropsychologia.

[19]  C. Kennard,et al.  Human Medial Frontal Cortex Mediates Unconscious Inhibition of Voluntary Action , 2007, Neuron.

[20]  Peter Hagoort,et al.  Beyond the sentence given , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[21]  Sandro Rubichi,et al.  Are visual stimuli sufficient to evoke motor information? Studies with hand primes , 2007, Neuroscience Letters.

[22]  G. Rizzolatti,et al.  Congruent Embodied Representations for Visually Presented Actions and Linguistic Phrases Describing Actions , 2006, Current Biology.

[23]  Mante S. Nieuwland,et al.  When Peanuts Fall in Love: N400 Evidence for the Power of Discourse , 2005, Journal of Cognitive Neuroscience.

[24]  A. Friederici,et al.  N400-like Semantic Incongruity Effect in 19-Month-Olds: Processing Known Words in Picture Contexts , 2004, Journal of Cognitive Neuroscience.

[25]  Dean F. Salisbury,et al.  Semantic memory and verbal working memory correlates of N400 to subordinate homographs , 2004, Brain and Cognition.

[26]  P. Hagoort,et al.  Integration of Word Meaning and World Knowledge in Language Comprehension , 2004, Science.

[27]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[28]  Hillel Pratt,et al.  A comparison of sentence- and discourse-level semantic processing: An ERP study , 2002, Brain and Language.

[29]  Blake W. Johnson,et al.  Comparison of the N300 and N400 ERPs to picture stimuli in congruent and incongruent contexts , 2002, Clinical Neurophysiology.

[30]  P. Holcomb,et al.  Event-related potentials during discourse-level semantic integration of complex pictures. , 2002, Brain research. Cognitive brain research.

[31]  W. Drees.  Explanatory Pluralism , 2001 .

[32]  Kara D. Federmeier,et al.  Meaning and modality: influences of context, semantic memory organization, and perceptual predictability on picture processing. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[33]  John F. Connolly,et al.  An event-related brain potential study of receptive speech comprehension using a modified Token Test , 1999, Neuropsychologia.

[34]  Colin M. Brown,et al.  Semantic Integration in Sentences and Discourse: Evidence from the N400 , 1999, Journal of Cognitive Neuroscience.

[35]  M. Kutas,et al.  The Search for Common Sense: An Electrophysiological Study of the Comprehension of Words and Pictures in Reading , 1996, Journal of Cognitive Neuroscience.

[36]  Julie C. Sedivy,et al.  Subject Terms: Linguistics Language Eyes & eyesight Cognition & reasoning , 1995 .

[37]  Julie C. Sedivy,et al.  Resolving attachment ambiguities with multiple constraints , 1995, Cognition.

[38]  J. Connolly,et al.  Event-Related Potential Components Reflect Phonological and Semantic Processing of the Terminal Word of Spoken Sentences , 1994, Journal of Cognitive Neuroscience.

[39]  P. Holcomb,et al.  Event-Related Brain Potentials Reflect Semantic Priming in an Object Decision Task , 1994, Brain and Cognition.

[40]  Shlomo Bentin,et al.  The effects of immediate stimulus repetition on reaction time and event-related potentials in tasks of different complexity , 1994 .

[41]  Peter Hagoort,et al.  The Processing Nature of the N400: Evidence from Masked Priming , 1993, Journal of Cognitive Neuroscience.

[42]  Nigam,et al.  N400 to Semantically Anomalous Pictures and Words , 1992, Journal of Cognitive Neuroscience.

[43]  M. Rugg.  Event-related brain potentials dissociate repetition effects of high-and low-frequency words , 1990, Memory & cognition.

[44]  J. Richard Jennings,et al.  Editorial Policy on Analyses of Variance With Repeated Measures , 1987 .

[45]  M. Potter,et al.  Pictures in sentences: understanding without words. , 1986, Journal of experimental psychology. General.

[46]  M. Rugg.  The effects of semantic priming and work repetition on event-related potentials. , 1985, Psychophysiology.

[47]  E Donchin,et al.  A new method for off-line removal of ocular artifact. , 1983, Electroencephalography and clinical neurophysiology.

[48]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[49]  L. M. M.-T..  Theory of Probability , 1929, Nature.