Learning by doing? The effect of gestures on implicit retrieval of newly acquired words

Meaningful gestures enhance speech comprehensibility. However, their role during novel-word acquisition remains elusive. Here we investigate how meaningful versus meaningless gestures impact on novel-word learning and contrast these conditions to a purely verbal training. After training, neuronal processing of the novel words was assessed by blood-oxygen-level-dependent functional magnetic resonance imaging (BOLD-fMRI), disclosing that networks affording retrieval differ depending on the training condition. Over 3 days participants learned pseudowords for common objects (e.g., /klira/ -cap). For training they repeated the novel word while performing (i) an iconic, (ii) a grooming or (iii) no gesture. For the two conditions involving gestures, these were either actively repeated or passively observed during training. Behaviorally no substantial differences between the five different training conditions were found while fMRI disclosed differential networks affording implicit retrieval of the learned pseudowords depending on the training procedure. Most notably training with actively performed iconic gestures yielded larger activation in a semantic network comprising left inferior frontal (BA47) and inferior temporal gyri. Additionally hippocampal activation was stronger for all trained compared to unknown pseudowords of identical structure. The behavioral results challenge the generality of an 'enactment-effect' for single word learning. Imaging results, however, suggest that actively performed meaningful gestures lead to a deeper semantic encoding of novel words. The findings are discussed regarding their implications for theoretical accounts and for empirical approaches of gesture-based strategies in language (re)learning.

[1]  Jacinta Douglas,et al.  The differential facilitatory effects of gesture and visualisation processes on object naming in aphasia , 2001 .

[2]  D. Poeppel,et al.  The cortical organization of speech processing , 2007, Nature Reviews Neuroscience.

[3]  B. Butterworth,et al.  Gesture, speech, and computational stages: a reply to McNeill. , 1989, Psychological review.

[4]  G. Winocur,et al.  The cognitive neuroscience of remote episodic, semantic and spatial memory , 2006, Current Opinion in Neurobiology.

[5]  A. Paivio,et al.  Dual coding theory and education , 1991 .

[6]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[7]  Roel M. Willems,et al.  Neural evidence for the interplay between language, gesture, and action: A review , 2007, Brain and Language.

[8]  Liuba Papeo,et al.  A cross-talk between brain-damage patients and infants on action and language , 2012, Neuropsychologia.

[9]  Willem J. M. Levelt,et al.  Pointing and voicing in deictic expressions , 1985 .

[10]  Sophie K Scott,et al.  The neural response to changing semantic and perceptual complexity during language processing , 2009, Human brain mapping.

[11]  Miranda L. Rose,et al.  The efficacy of repetition, semantic, and gesture treatments for verb retrieval and use in Broca's aphasia , 2011 .

[12]  Qiyong Gong,et al.  Robust and Task-Independent Spatial Profile of the Visual Word Form Activation in Fusiform Cortex , 2011, PloS one.

[13]  R. L. Cohen,et al.  On the generality of some memory laws , 1981 .

[14]  Dare A. Baldwin,et al.  Establishing word-object relations: a first step. , 1989, Child development.

[15]  Richard S. J. Frackowiak,et al.  Functional anatomy of a common semantic system for words and pictures , 1996, Nature.

[16]  E. Tulving,et al.  Hippocampal PET activations of memory encoding and retrieval: The HIPER model , 1998, Hippocampus.

[17]  A. Friederici,et al.  The impact of iconic gestures on foreign language word learning and its neural substrate , 2011, Human brain mapping.

[18]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[19]  William W. Graves,et al.  Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. , 2009, Cerebral cortex.

[20]  W. Gill Woodall,et al.  Nonverbal cue context and episodic memory: On the availability and endurance of nonverbal behaviors as retrieval cues , 1985 .

[21]  Keith E. Nelson,et al.  Concepts and words in the 18-month-old: Acquiring concept names under controlled conditions , 1973 .

[22]  Nicoletta Caramelli,et al.  Behavioural and neurophysiological evidence of semantic interaction between iconic gestures and words , 2008, Cognitive neuropsychology.

[23]  S. Ravizza,et al.  Movement and lexical access: Do noniconic gestures aid in retrieval? , 2003, Psychonomic bulletin & review.

[24]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[25]  T. Talavage,et al.  Neuroanatomical distribution of five semantic components of verbs: Evidence from fMRI , 2008, Brain and Language.

[26]  Pierre Feyereisen,et al.  Enactment effects and integration processes in younger and older adults’ memory for actions , 2009, Memory.

[27]  J. Bower,et al.  Involvement of the cerebellum in semantic discrimination: An fMRI study , 2003, Human brain mapping.

[28]  C. Price The anatomy of language: a review of 100 fMRI studies published in 2009 , 2010, Annals of the New York Academy of Sciences.

[29]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[30]  Isabell Wartenburger,et al.  Implicit Processing of Phonotactic Cues: Evidence from Electrophysiological and Vascular Responses , 2011, Journal of Cognitive Neuroscience.

[31]  J. Desmond,et al.  Neuroimaging studies of the cerebellum: language, learning and memory , 1998, Trends in Cognitive Sciences.

[32]  Gereon R. Fink,et al.  Action verbs and the primary motor cortex: A comparative TMS study of silent reading, frequency judgments, and motor imagery , 2008, Neuropsychologia.

[33]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[34]  M. Corballis Mirror neurons and the evolution of language , 2010, Brain and Language.

[35]  M. Corballis,et al.  From manual gesture to speech: A gradual transition , 2006, Neuroscience & Biobehavioral Reviews.

[36]  M. Posner,et al.  Positron Emission Tomographic Studies of the Processing of Singe Words , 1989, Journal of Cognitive Neuroscience.

[37]  Miranda Rose,et al.  The utility of arm and hand gestures in the treatment of aphasia , 2006 .

[38]  J. Kaiser,et al.  Object Familiarity and Semantic Congruency Modulate Responses in Cortical Audiovisual Integration Areas , 2007, The Journal of Neuroscience.

[39]  David Badre,et al.  Left ventrolateral prefrontal cortex and the cognitive control of memory , 2007, Neuropsychologia.

[40]  Angela D. Friederici,et al.  Neural Correlates of High Performance in Foreign Language Vocabulary Learning. , 2010 .

[41]  L. Allen The Effects of Emblematic Gestures on the Development and Access of Mental Representations of French Expressions , 1995 .

[42]  Ulf Liszkowski,et al.  15-Month-Old Infants Fast Map Words but Not Representational Gestures of Multimodal Labels , 2012, Front. Psychology.

[43]  A. Nobre,et al.  Dissociating Linguistic Processes in the Left Inferior Frontal Cortex with Transcranial Magnetic Stimulation , 2022 .

[44]  Margaret Gwendoline Riseborough,et al.  Physiographic gestures as decoding facilitators: Three experiments exploring a neglected facet of communication , 1981 .

[45]  L. Fadiga,et al.  Lexicality drives audio-motor transformations in Broca’s area , 2010, Brain and Language.

[46]  D. McNeill Hand and Mind: What Gestures Reveal about Thought , 1992 .

[47]  Roberto Cabeza,et al.  Neural Correlates of Relational Memory: Successful Encoding and Retrieval of Semantic and Perceptual Associations , 2005, The Journal of Neuroscience.

[48]  R. C. Oldfield The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[49]  G. Beattie,et al.  An experimental investigation of the role of iconic gestures in lexical access using the tip-of-the-tongue phenomenon. , 1999, British journal of psychology.

[50]  Thomas Wolbers,et al.  Hippocampus activity differentiates good from poor learners of a novel lexicon , 2005, NeuroImage.

[51]  S. Kelly,et al.  Please Scroll down for Article Language and Cognitive Processes Brief Training with Co-speech Gesture Lends a Hand to Word Learning in a Foreign Language , 2022 .

[52]  N Burgess,et al.  Recoding, storage, rehearsal and grouping in verbal short-term memory: an fMRI study , 2000, Neuropsychologia.

[53]  Antoni Rodríguez-Fornells,et al.  Functional Neuroanatomy of Meaning Acquisition from Context , 2008, Journal of Cognitive Neuroscience.

[54]  Prahlad Gupta,et al.  Word learning, phonological short-term memory, phonotactic probability and long-term memory: towards an integrated framework , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[55]  Antonino Vallesi,et al.  Effects of TMS on Different Stages of Motor and Non-Motor Verb Processing in the Primary Motor Cortex , 2009, PloS one.

[56]  Susan M. Wagner,et al.  How our hands help us learn , 2005, Trends in Cognitive Sciences.

[57]  R. Krauss,et al.  The Role of Speech-Related Arm/Hand Gestures in Word Retrieval , 2001 .

[58]  Richard S. J. Frackowiak,et al.  The neural correlates of the verbal component of working memory , 1993, Nature.

[59]  Tilo Kircher,et al.  Memory Effects of Speech and Gesture Binding: Cortical and Hippocampal Activation in Relation to Subsequent Memory Performance , 2009, Journal of Cognitive Neuroscience.

[60]  Andreas Hennenlotter,et al.  Neural correlates of the processing of co-speech gestures , 2008, NeuroImage.

[61]  Michael R Brent,et al.  Reexamining the vocabulary spurt. , 2004, Developmental psychology.

[62]  L. Markson,et al.  Memory for Visual-Spoken Language in Children and Adults , 1998 .

[63]  J. D. Ruiter The production of gesture and speech , 2000 .

[64]  A. Jansen,et al.  Neural integration of iconic and unrelated coverbal gestures: A functional MRI study , 2009, Human brain mapping.

[65]  Matthew H. Davis,et al.  A complementary systems account of word learning: neural and behavioural evidence , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[66]  E. Tulving,et al.  Episodic and declarative memory: Role of the hippocampus , 1998, Hippocampus.

[67]  Pierre Feyereisen,et al.  Further investigation on the mnemonic effect of gestures: Their meaning matters , 2006 .

[68]  Jeremy I. Skipper,et al.  Speech-associated gestures, Broca’s area, and the human mirror system , 2007, Brain and Language.

[69]  James L. McClelland,et al.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. , 1995, Psychological review.

[70]  Ana B. Chica,et al.  Attentional Routes to Conscious Perception , 2012, Front. Psychology.

[71]  Gereon R. Fink,et al.  Stimulus properties matter more than perspective: An fMRI study of mental imagery and silent reading of action phrases , 2007, NeuroImage.

[72]  Matthew H. Davis,et al.  Learning and Consolidation of Novel Spoken Words , 2009, Journal of Cognitive Neuroscience.

[73]  Cristina Burani,et al.  Supercalifragilisticexpialidocious: How the brain learns words never heard before , 2009, NeuroImage.

[74]  M. Tellier The effect of gestures on second language memorisation by young children , 2008 .

[75]  Roel M. Willems,et al.  When language meets action: the neural integration of gesture and speech. , 2007, Cerebral cortex.

[76]  Emily B. Myers,et al.  Recruitment of anterior and posterior structures in lexical–semantic processing: An fMRI study comparing implicit and explicit tasks , 2008, Brain and Language.

[77]  P. Hagoort,et al.  Language beyond action , 2008, Journal of Physiology-Paris.

[78]  Liuba Papeo,et al.  Action performance and action-word understanding: Evidence of double dissociations in left-damaged patients , 2010, Cognitive neuropsychology.

[79]  A. Friederici The brain basis of language processing: from structure to function. , 2011, Physiological reviews.

[80]  A. Paivio,et al.  Concrete image and verbal memory codes , 1969 .

[81]  Elizabeth Jefferies,et al.  Heterogeneity of the Left Temporal Lobe in Semantic Representation and Control: Priming Multiple versus Single Meanings of Ambiguous Words , 2010, Cerebral cortex.