Co-speech iconic gestures and visuo-spatial working memory.

Three experiments tested the role of verbal versus visuo-spatial working memory in the comprehension of co-speech iconic gestures. In Experiment 1, participants viewed congruent discourse primes in which the speaker's gestures matched the information conveyed by his speech, and incongruent ones in which the semantic content of the speaker's gestures diverged from that in his speech. Discourse primes were followed by picture probes that participants judged as being either related or unrelated to the preceding clip. Performance on this picture probe classification task was faster and more accurate after congruent than incongruent discourse primes. The effect of discourse congruency on response times was linearly related to measures of visuo-spatial, but not verbal, working memory capacity, as participants with greater visuo-spatial WM capacity benefited more from congruent gestures. In Experiments 2 and 3, participants performed the same picture probe classification task under conditions of high and low loads on concurrent visuo-spatial (Experiment 2) and verbal (Experiment 3) memory tasks. Effects of discourse congruency and verbal WM load were additive, while effects of discourse congruency and visuo-spatial WM load were interactive. Results suggest that congruent co-speech gestures facilitate multi-modal language comprehension, and indicate an important role for visuo-spatial WM in these speech-gesture integration processes.

[1]  Geoffrey Beattie,et al.  An experimental investigation of the role of different types of iconic gesture in communication: A semantic feature approach , 2003 .

[2]  Martha W. Alibali,et al.  Children’s gestures are meant to be seen , 2002 .

[3]  Tilo Kircher,et al.  A Supramodal Neural Network for Speech and Gesture Semantics: An fMRI Study , 2012, PloS one.

[4]  P. Holcomb,et al.  Event-Related Brain Potentials Reflect Semantic Priming in an Object Decision Task , 1994, Brain and Cognition.

[5]  Michael F. Bunting,et al.  Working memory span tasks: A methodological review and user’s guide , 2005, Psychonomic bulletin & review.

[6]  P. Carpenter,et al.  Individual differences in working memory and reading , 1980 .

[7]  Ying Choon Wu,et al.  Meaningful gestures: electrophysiological indices of iconic gesture comprehension. , 2005, Psychophysiology.

[8]  James E. Hoffman,et al.  Individual Differences in Inference Generation: An ERP Analysis , 1997, Journal of Cognitive Neuroscience.

[9]  A. Miyake,et al.  The separability of working memory resources for spatial thinking and language processing: an individual differences approach. , 1996, Journal of experimental psychology. General.

[10]  D. Berch,et al.  The Corsi Block-Tapping Task: Methodological and Theoretical Considerations , 1998, Brain and Cognition.

[11]  R. Krauss,et al.  The Communicative Value of Conversational Hand Gesture , 1995 .

[12]  Raymond W. Gibbs,et al.  Emotional Reactions to Verbal Irony , 2000 .

[13]  M. D’Esposito.  Working memory. , 2008, Handbook of clinical neurology.

[14]  Stuart Rosen,et al.  The Neural Basis of Cross Modal Enhancement in Speech Perception: A Pet Study , 2002 .

[15]  Jeremy J Tree,et al.  Sometimes faster, sometimes slower: associative and competitor priming in picture naming with young and elderly participants , 2003, Journal of Neurolinguistics.

[16]  Martha W. Alibali,et al.  Cognitive skills and gesture–speech redundancy: Formulation difficulty or communicative strategy? , 2011 .

[17]  K. Emmorey,et al.  Gesture, Thought, and Spatial Language , 2001 .

[18]  R. Klatzky,et al.  Teachers’ gestures facilitate students’ learning: A lesson in symmetry , 2003 .

[19]  S. Goldin-Meadow,et al.  Learning what children know about space from looking at their hands: the added value of gesture in spatial communication. , 2012, Journal of experimental child psychology.

[20]  B. Milner,et al.  Interhemispheric differences in the localization of psychological processes in man. , 1971, British medical bulletin.

[21]  Michael P. Kaschak,et al.  Putting words in perspective , 2004, Memory & cognition.

[22]  J. Jonides,et al.  On the Cost and Benefit of Cost and Benefit , 1984 .

[23]  S. Kelly,et al.  Please Scroll down for Article Language and Cognitive Processes Brief Training with Co-speech Gesture Lends a Hand to Word Learning in a Foreign Language , 2022 .

[24]  Susan M. Wagner,et al.  Probing the Mental Representation of Gesture: Is Handwaving Spatial?. , 2004 .

[25]  Susan Goldin-Meadow,et al.  Children Learn When Their Teacher's Gestures and Speech Differ , 2005, Psychological science.

[26]  G. Waters,et al.  The Measurement of Verbal Working Memory Capacity and Its Relation to Reading Comprehension , 1996, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[27]  Martha W. Alibali,et al.  The Role of Gesture in Children's Comprehension of Spoken Language:Now They Need It, Now They Don't , 2000 .

[28]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[29]  Stacy B. Ehrlich,et al.  The importance of gesture in children's spatial reasoning. , 2006, Developmental psychology.

[30]  S. Kelly,et al.  Neural correlates of bimodal speech and gesture comprehension , 2004, Brain and Language.

[31]  Sotaro Kita,et al.  Spontaneous Gestures Influence Strategy Choices in Problem Solving , 2011, Psychological science.

[32]  Martha W. Alibali,et al.  Gesture in Spatial Cognition: Expressing, Communicating, and Thinking About Spatial Information , 2005, Spatial Cogn. Comput..

[33]  Ying Choon Wu,et al.  How iconic gestures enhance communication: An ERP study , 2007, Brain and Language.

[34]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[35]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[36]  Seana Coulson,et al.  Gestures modulate speech processing early in utterances , 2010, Neuroreport.

[37]  Uri Hadar,et al.  Iconic gestures: the grammatical categories of lexical affiliates , 1999, Journal of Neurolinguistics.

[38]  S. Goldin-Meadow,et al.  Making children gesture brings out implicit knowledge and leads to learning. , 2007, Journal of experimental psychology. General.

[39]  James Bartolotti,et al.  Integrating Speech and Iconic Gestures in a Stroop-like Task: Evidence for Automatic Processing , 2010, Journal of Cognitive Neuroscience.

[40]  Arthur M. Glenberg,et al.  Indexical understanding of instructions , 1999 .

[41]  M. Argyle,et al.  A Cross-Cultural Study of the Communication of Extra-Verbal Meaning by Gesture , 1975 .

[42]  Sotaro Kita,et al.  On-line Integration of Semantic Information from Speech and Gesture: Insights from Event-related Brain Potentials , 2007, Journal of Cognitive Neuroscience.

[43]  William D. Hopkins,et al.  The effect of thought structure on the production of lexical movements , 2002, Brain and Language.

[44]  Shahrzad Mahootian,et al.  The Role of Gesture in Bilingual Education: Does Gesture Enhance Learning? , 2004 .

[45]  Elizabeth S. Spelke,et al.  Sources of Flexibility in Human Cognition: Dual-Task Studies of Space and Language , 1999, Cognitive Psychology.

[46]  Ying Choon Wu,et al.  Are depictive gestures like pictures? Commonalities and differences in semantic processing , 2011, Brain and Language.

[47]  S. Kita,et al.  Individual Differences in Frequency and Saliency of Speech-Accompanying Gestures: The Role of Cognitive Abilities and Empathy , 2013, Journal of experimental psychology. General.

[48]  D. McNeill.  Hand and Mind , 1995 .

[49]  De Ruiter,et al.  Gesture and speech production , 1998 .

[50]  Nicoletta Caramelli,et al.  Behavioural and neurophysiological evidence of semantic interaction between iconic gestures and words , 2008, Cognitive neuropsychology.

[51]  Thomas C. Gunter,et al.  The Role of Iconic Gestures in Speech Disambiguation: ERP Evidence , 2007, Journal of Cognitive Neuroscience.

[52]  Christopher D. Wickens,et al.  The Structure of Attentional Resources , 1980 .

[53]  J. Palmer.  Set-size effects in visual search: The effect of attention is independent of the stimulus for simple tasks , 1994, Vision Research.

[54]  D. Kimura,et al.  Hand movement asymmetry during speech: No effect of speaking topic , 1987, Neuropsychologia.

[55]  Yukari Hirata,et al.  Effects of lips and hands on auditory learning of second-language speech sounds. , 2010, Journal of speech, language, and hearing research : JSLHR.

[56]  M. Alibali,et al.  Effects of Visibility between Speaker and Listener on Gesture Production: Some Gestures Are Meant to Be Seen , 2001 .

[57]  Rolf A. Zwaan,et al.  Language Comprehenders Mentally Represent the Shapes of Objects , 2002, Psychological science.

[58]  M. Rugg,et al.  Event-related potentials and the semantic matching of pictures , 1990, Brain and Cognition.

[59]  Martha W. Alibali,et al.  Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production , 2007 .

[60]  Ezequiel Morsella,et al.  The role of gestures in spatial working memory and speech. , 2004, The American journal of psychology.

[61]  J. Segui,et al.  Semantic and Associative Priming in Picture Naming , 2000, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[62]  A. Baddeley.  Working memory: looking back and looking forward , 2003, Nature Reviews Neuroscience.

[63]  R Wesp,et al.  Gestures maintain spatial imagery. , 2001, The American journal of psychology.

[64]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[65]  S. Goldin-Meadow,et al.  Gesturing makes learning last , 2008, Cognition.

[66]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.