The role of iconic gestures in production and comprehension of language: evidence from brain and behavior

Speakers in all cultures and ages use gestures as they speak (i.e., cospeech gestures). There have been different views in the literature with regard to whether and how a specific type of gestures speakers use, i.e., iconic gestures, interacts with language processing. Here I review evidence showing that iconic gestures are not produced merely from the spatial and/or motoric imagery but from an in interface representation of imagistic and linguistic representation during online speaking Similarly, for comprehension, neuroimaging and behavioral studies indicate that speech and gesture influences semantic processing of each other during online comprehension. These findings show overall that processing of information in both modalities interacts during both comprehension and production of language arguing against models that propose independent processing of each modality. They also have implications for AI models that aim to simulate cospeech gesture use in conversational agents.

[1]  G. Calvert.  Crossmodal processing in the human brain: insights from functional neuroimaging studies. , 2001, Cerebral cortex.

[2]  Leonard Talmy,et al.  Semantics and syntax of motion , 1975 .

[3]  Rachel I. Mayberry,et al.  Language and Gesture: Gesture production during stuttered speech: insights into the nature of gesture–speech integration , 2000 .

[4]  R. Krauss,et al.  Do conversational hand gestures communicate? , 1991, Journal of personality and social psychology.

[5]  Janet Beavin Bavelas,et al.  Gesturing on the telephone: Independent effects of dialogue and visibility. , 2008 .

[6]  Autumn B. Hostetter,et al.  Visible embodiment: Gestures as simulated action , 2008, Psychonomic bulletin & review.

[7]  Sotaro Kita,et al.  How does linguistic framing of events influence co-speech gestures? , 2007 .

[8]  T. Shopen,et al.  Language typology and syntactic description , 2013 .

[9]  GEOFFREY BEATTIE,et al.  Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation , 1999 .

[10]  E. Maris,et al.  Two Sides of the Same Coin , 2010, Psychological science.

[11]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[12]  David P. Wilkins,et al.  Semantic typology and spatial conceptualization , 1998 .

[13]  Kathryn Bock,et al.  Toward a Cognitive Psychology of Syntax: Information Processing Contributions to Sentence Formulation , 1982 .

[14]  Sotaro Kita,et al.  On-line Integration of Semantic Information from Speech and Gesture: Insights from Event-related Brain Potentials , 2007, Journal of Cognitive Neuroscience.

[15]  Sotaro Kita,et al.  Expressing manner and path in English and Turkish: Differences in speech, gesture, and conceptualization , 2020, Proceedings of the Twenty First Annual Conference of the Cognitive Science Society.

[16]  Asli Ozyurek.  Do Speakers Design Their Cospeech Gestures for Their Addressees? The Effects of Addressee Location on Representational Gestures , 2002 .

[17]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.

[18]  S. Stoness,et al.  Proceedings of the Twenty First Annual Conference of the Cognitive Science Society , 1999 .

[19]  D. McNeill.  Language and Gesture , 2000 .

[20]  J. Lannoy,et al.  Gestures and Speech: Psychological Investigations , 1991 .

[21]  Peter Hagoort,et al.  How the brain solves the binding problem for language: a neurocomputational model of syntactic processing , 2003, NeuroImage.

[22]  D. McNeill.  So you think gestures are nonverbal , 1985 .

[23]  D. McNeill.  Hand and Mind , 1995 .

[24]  J. D. Ruiter.  The production of gesture and speech , 2000 .

[25]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[26]  Sotaro Kita,et al.  Relations between syntactic encoding and co-speech gestures: Implications for a model of speech and gesture production , 2007 .

[27]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[28]  P. Hagoort,et al.  Integration of Word Meaning and World Knowledge in Language Comprehension , 2004, Science.

[29]  Sotaro Kita,et al.  How does linguistic framing influence co-speech gestures? Insights from crosslinguistic differences and similarities , 2007 .

[30]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .