High contextual sensitivity of metaphorical expressions and gesture blending: A video event-related potential design

Human communication in a natural context implies the dynamic coordination of contextual clues, paralinguistic information and literal as well as figurative language use. In the present study we constructed a paradigm with four types of video clips: literal and metaphorical expressions accompanied by congruent and incongruent gesture actions. Participants were instructed to classify the gesture accompanying the expression as congruent or incongruent by pressing two different keys while electrophysiological activity was being recorded. We compared behavioral measures and event related potential (ERP) differences triggered by the gesture stroke onset. Accuracy data showed that incongruent metaphorical expressions were more difficult to classify. Reaction times were modulated by incongruent gestures, by metaphorical expressions and by a gesture-expression interaction. No behavioral differences were found between the literal and metaphorical expressions when the gesture was congruent. N400-like and LPC-like (late positive complex) components from metaphorical expressions produced greater negativity. The N400-like modulation of metaphorical expressions showed a greater difference between congruent and incongruent categories over the left anterior region, compared with the literal expressions. More importantly, the literal congruent as well as the metaphorical congruent categories did not show any difference. Accuracy, reaction times and ERPs provide convergent support for a greater contextual sensitivity of the metaphorical expressions.

[1]  C. Petten,et al.  Conceptual integration and metaphor: An event-related potential study , 2002, Memory & cognition.

[2]  D P Salmon,et al.  Abnormal verbal event related potentials in mild cognitive impairment and incipient Alzheimer's disease , 2002, Journal of neurology, neurosurgery, and psychiatry.

[3]  Willem J. M. Levelt,et al.  Pointing and voicing in deictic expressions , 1985 .

[4]  Roel M. Willems,et al.  Neural evidence for the interplay between language, gesture, and action: A review , 2007, Brain and Language.

[5]  J. Mahon.  The Poetics of Mind , 1996 .

[6]  Asli Ozyurek,et al.  Gesture, language, and brain , 2007 .

[7]  Thomas C. Gunter,et al.  The Role of Iconic Gestures in Speech Disambiguation: ERP Evidence , 2007, Journal of Cognitive Neuroscience.

[8]  R. Giora.  On Our Mind: Salience, Context, and Figurative Language , 2003 .

[9]  P. Semm,et al.  Electrophysiology of the guinea-pig pineal organ: Sympathetically influenced cells responding differently to light and darkness , 1979, Neuroscience Letters.

[10]  P. Holcomb,et al.  Event-related potentials during discourse-level semantic integration of complex pictures. , 2002, Brain research. Cognitive brain research.

[11]  Vincent M. Reid,et al.  N400 involvement in the processing of action sequences , 2008, Neuroscience Letters.

[12]  J. Pynte,et al.  The Time-Course of Metaphor Comprehension: An Event-Related Potential Study , 1996, Brain and Language.

[13]  F. Manes,et al.  Exploring the relationship between vagal tone and event-related potentials in response to an affective picture task , 2011, Social neuroscience.

[14]  U. Bellugi,et al.  Neural Systems Mediating American Sign Language: Effects of Sensory Experience and Age of Acquisition , 1997, Brain and Language.

[15]  Bo Ekman.  In Search of Common Sense , 2007 .

[16]  Alice Mado Proverbio,et al.  RP and N400 ERP components reflect semantic violations in visual processing of human actions , 2009, Neuroscience Letters.

[17]  D. Friedman.  Cognition and Aging: A Highly Selective Overview of Event-Related Potential (ERP) Data , 2003, Journal of clinical and experimental neuropsychology.

[18]  Sotaro Kita,et al.  On-line Integration of Semantic Information from Speech and Gesture: Insights from Event-related Brain Potentials , 2007, Journal of Cognitive Neuroscience.

[19]  Roel M. Willems,et al.  Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language , 2009, NeuroImage.

[20]  M. Kutas,et al.  An electrophysiological study of scene effects on object identification. , 2003, Brain research. Cognitive brain research.

[21]  Kara D. Federmeier,et al.  Meaning and modality: influences of context, semantic memory organization, and perceptual predictability on picture processing. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[22]  C. Braun,et al.  Speech rate as a sticky switch: A multiple lesion case analysis of mutism and hyperlalia , 2004, Brain and Language.

[23]  Spencer D. Kelly,et al.  Gesture, brain, and language , 2007, Brain and Language.

[24]  Phillip J Holcomb,et al.  Semantic integration in videos of real-world events: an electrophysiological investigation. , 2003, Psychophysiology.

[25]  A. Ibanez,et al.  Multi-level analysis of cultural phenomena: The role of ERPs approach to prejudice , 2009 .

[26]  P. Holcomb,et al.  An electrophysiological investigation of semantic priming with pictures of real objects. , 1999, Psychophysiology.

[27]  S. Coulson.  Constructing Meaning , 2006 .

[28]  A. Ibanez,et al.  The Machine Paradigm and Alternative Approaches in Cognitive Science , 2010, Integrative psychological & behavioral science.

[29]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[30]  Patric Bach,et al.  Communicating hands: ERPs elicited by meaningful symbolic hand postures , 2004, Neuroscience Letters.

[31]  Neal J. Cohen,et al.  Event-related Potential Signatures of Relational Memory , 2006, Journal of Cognitive Neuroscience.

[32]  M. Kutas,et al.  Reading senseless sentences: brain potentials reflect semantic incongruity. , 1980, Science.

[33]  Ying Choon Wu,et al.  How iconic gestures enhance communication: An ERP study , 2007, Brain and Language.

[34]  A. Ibanez,et al.  Moving Beyond Computational Cognitivism: Understanding Intentionality, Intersubjectivity and Ecology of Mind , 2008, Integrative psychological & behavioral science.

[35]  B Ska,et al.  Conceptual apraxia and semantic memory deficit in Alzheimer's disease: Two sides of the same coin? , 2000, Journal of the International Neuropsychological Society.

[36]  M. Kutas,et al.  The Search for Common Sense: An Electrophysiological Study of the Comprehension of Words and Pictures in Reading , 1996, Journal of Cognitive Neuroscience.

[37]  Diego Cosmelli,et al.  Human Cognition in Context: On the Biologic, Cognitive and Social Reconsideration of Meaning as Making Sense of Action , 2008, Integrative psychological & behavioral science.

[38]  Carlos Cornejo.  Who Says What the Words Say? , 2004 .

[39]  Barbara Tillmann,et al.  Temporal Aspects of the Feeling of Familiarity for Music and the Emergence of Conceptual Processing , 2010, Journal of Cognitive Neuroscience.

[40]  Ying Choon Wu,et al.  Meaningful gestures: electrophysiological indices of iconic gesture comprehension. , 2005, Psychophysiology.

[41]  A. Ibanez,et al.  N400 deficits from semantic matching of pictures in probands and first-degree relatives from multiplex schizophrenia families , 2009, Brain and Cognition.

[42]  R. Gibbs.  The Poetics of Mind: Figurative Thought, Language, and Understanding , 1994 .

[43]  Vladimir López,et al.  ERPs and contextual semantic discrimination: Degrees of congruence in wakefulness and sleep , 2006, Brain and Language.

[44]  Agustín Ibáñez,et al.  Size and probability of rewards modulate the feedback error-related negativity associated with wins but not losses in a monetarily rewarded gambling task , 2010, NeuroImage.

[45]  F. Manes,et al.  Applauding with Closed Hands: Neural Signature of Action-Sentence Compatibility Effects , 2010, PloS one.

[46]  F. Manes,et al.  When context is difficult to integrate: Cortical measures of congruency in schizophrenics and healthy relatives from multiplex families , 2011, Schizophrenia Research.

[47]  Facundo Manes,et al.  Contextual blending of ingroup/outgroup face stimuli and word valence: LPP modulation and convergence of measures , 2009, BMC Neuroscience.

[48]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.

[49]  B. Oken,et al.  Statistical issues concerning computerized analysis of brainwave topography , 1986, Annals of neurology.

[50]  M. Rugg,et al.  Event-related potentials and the semantic matching of pictures , 1990, Brain and Cognition.

[51]  Franco Simonetti,et al.  Gesture and metaphor comprehension: Electrophysiological evidence of cross-modal coordination by audiovisual stimulation , 2009, Brain and Cognition.

[52]  K. Willmes,et al.  Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures , 2009, Neuropsychologia.

[53]  S. Kelly,et al.  Neural correlates of bimodal speech and gesture comprehension , 2004, Brain and Language.

[54]  James Bartolotti,et al.  An intentional stance modulates the integration of gesture and speech during comprehension , 2007, Brain and Language.

[55]  Carlos Cornejo.  Review Essay: Conceptualizing Metaphors versus Embodying the Language , 2007 .