Hand Gesture and Mathematics Learning: Lessons From an Avatar

A beneficial effect of gesture on learning has been demonstrated in multiple domains, including mathematics, science, and foreign language vocabulary. However, because gesture is known to co-vary with other non-verbal behaviors, including eye gaze and prosody along with face, lip, and body movements, it is possible the beneficial effect of gesture is instead attributable to these other behaviors. We used a computer-generated animated pedagogical agent to control both verbal and non-verbal behavior. Children viewed lessons on mathematical equivalence in which an avatar either gestured or did not gesture, while eye gaze, head position, and lip movements remained identical across gesture conditions. Children who observed the gesturing avatar learned more, and they solved problems more quickly. Moreover, those children who learned were more likely to transfer and generalize their knowledge. These findings provide converging evidence that gesture facilitates math learning, and they reveal the potential for using technology to study non-verbal behavior in controlled experiments.

[1]  Jeeheon Ryu,et al.  The Effects of Image and Animation in Enhancing Pedagogical Agent Persona , 2003 .

[2]  Scotty D. Craig,et al.  Animated Pedagogical Agents in Multimedia Educational Environments: Effects of Agent Properties, Picture Features, and Redundancy , 2002 .

[3]  Martin Reisslein,et al.  Using Virtual Peers to Guide Visual Attention During Learning , 2010, J. Media Psychol. Theor. Methods Appl..

[4]  R. Atkinson Optimizing learning from examples using animated pedagogical agents. , 2002 .

[5]  S. Goldin-Meadow,et al.  What the teacher's hands tell the student's mind about math. , 1999 .

[6]  Slava Kalyuga,et al.  Managing split-attention and redundancy in multimedia instruction , 1999 .

[7]  R. Mayer The Cambridge Handbook of Multimedia Learning: Principles of Multimedia Learning Based on Social Cues : Personalization, Voice, and Image Principles , 2005 .

[8]  J. Titchener Experimenter Effects in Behavioral Research. , 1967 .

[9]  Soyoung Kim,et al.  Designing nonverbal communication for pedagogical agents: When less is more , 2009, Comput. Hum. Behav..

[10]  Matthew Flatt,et al.  PsyScope: An interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers , 1993 .

[11]  Shahrzad Mahootian,et al.  The Role of Gesture in Bilingual Education: Does Gesture Enhance Learning? , 2004 .

[12]  R. Klatzky,et al.  Teachers’ gestures facilitate students’ learning: A lesson in symmetry , 2003 .

[13]  F. Paas,et al.  Towards a Framework for Attention Cueing in Instructional Animations: Guidelines for Research and Design , 2009 .

[14]  Rainer Bromme,et al.  Coherence Formation when Learning from Text and Pictures: What Kind of Support for Whom? , 2009 .

[15]  Erol Özçelik,et al.  Why does signaling enhance multimedia learning? Evidence from eye movements , 2010, Comput. Hum. Behav..

[16]  Kristinn R. Thórisson,et al.  The Power of a Nod and a Glance: Envelope Vs. Emotional Feedback in Animated Conversational Agents , 1999, Appl. Artif. Intell..

[17]  Thomas Huk,et al.  The educational value of visual cues and 3D-representational format in a computer animation under restricted and realistic conditions , 2010 .

[18]  Elisabeth André,et al.  The Persona Effect: How Substantial Is It? , 1998, BCS HCI.

[19]  Rachel Barr,et al.  Transfer of learning between 2D and 3D sources during infancy: Informing theory and practice. , 2010, Developmental review : DR.

[20]  K S Yen [Conversing with children]. , 1971, Hu li za zhi The journal of nursing.

[21]  Mitchell J. Nathan,et al.  Embodiment in Mathematics Teaching and Learning: Evidence From Learners' and Teachers' Gestures , 2012 .

[22]  Abraham Arcavi,et al.  The role of visual representations in the learning of mathematics , 2003 .

[23]  R. Mayer,et al.  Interactive Multimodal Learning Environments , 2007 .

[24]  Susan M. Wagner,et al.  Explaining Math: Gesturing Lightens the Load , 2001, Psychological science.

[25]  H. Tabbers,et al.  Multimedia instructions and cognitive load theory: effects of modality and cueing. , 2004, The British journal of educational psychology.

[26]  Stacy Marsella,et al.  How to Train Your Avatar: A Data Driven Approach to Gesture Generation , 2011, IVA.

[27]  Max M. Louwerse,et al.  Effects of Ambiguous Gestures and Language on the Time Course of Reference Resolution , 2010, Cogn. Sci..

[28]  S. Goldin-Meadow,et al.  The Role of Gesture in Learning: Do Children Use Their Hands to Change Their Minds? , 2006 .

[29]  Ronan S. Bernas,et al.  Engaging ADHD students in tasks with hand gestures: a pedagogical possibility for teachers , 2004 .

[30]  Monica Bucciarelli,et al.  The Deep Versus the Shallow: Effects of Co-Speech Gestures in Learning From Discourse , 2008, Cogn. Sci..

[31]  D. Barr,et al.  Random effects structure for confirmatory hypothesis testing: Keep it maximal. , 2013, Journal of memory and language.

[32]  M. Rowe,et al.  The role of pictures and gestures as nonverbal aids in preschoolers’ word learning in a novel language , 2013 .

[33]  M. Malbrán The Cambridge Handbook of Multimedia Learning , 2007 .

[34]  Shrikanth S. Narayanan,et al.  Comparison of child-human and child-computer interactions based on manual annotations , 2009, WOCCI '09.

[35]  Karl F. MacDorman,et al.  The Uncanny Valley [From the Field] , 2012, IEEE Robotics Autom. Mag..

[36]  Amy L. Baylor,et al.  The impact of three pedagogical agent roles , 2003, AAMAS '03.

[37]  Martha W. Alibali,et al.  You'll see what you mean: Students encode equations based on their knowledge of arithmetic , 2004, Cogn. Sci..

[38]  Melissa Singer,et al.  The Role of Gesture in Meaning Construction , 2008 .

[39]  Björn B. de Koning,et al.  Gestures in Instructional Animations: A Helping Hand to Understanding Non-human Movements? , 2013 .

[40]  Robert K. Atkinson,et al.  Animated pedagogical agents: does their degree of embodiment impact learning from static or animated worked examples? , 2007 .

[41]  Stefan Kopp,et al.  Individualized Gesturing Outperforms Average Gesturing - Evaluating Gesture Production in Virtual Humans , 2010, IVA.

[42]  Robert K. Atkinson,et al.  Fostering multimedia learning of science: Exploring the role of an animated agent's image , 2007, Comput. Educ..

[43]  James C. Lester,et al.  The persona effect: affective impact of animated pedagogical agents , 1997, CHI.

[44]  Percival G. Matthews,et al.  Measure for Measure: what Combining Diverse Measures Reveals About Children's Understanding of the Equal Sign as an Indicator of Mathematical Equality , 2012 .

[45]  G. Clarebout,et al.  Do pedagogical agents make a difference to student motivation and learning , 2011 .

[46]  Susan Goldin-Meadow,et al.  Children Learn When Their Teacher's Gestures and Speech Differ , 2005, Psychological science.

[47]  Kirsten Bergmann,et al.  A Virtual Agent as Vocabulary Trainer: Iconic Gestures Help to Improve Learners' Memory Performance , 2013, IVA.

[48]  Susan Goldin-Meadow,et al.  Do gestures communicate , 2005 .

[49]  Jean-Claude Martin,et al.  The effects of speech-gesture cooperation in animated agents' behavior in multimedia presentations , 2007, Interact. Comput..

[50]  A. Friederici,et al.  The impact of iconic gestures on foreign language word learning and its neural substrate , 2011, Human brain mapping.

[51]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[52]  James C. Lester,et al.  Animated pedagogical agents and problem-solving effectiveness: a large-scale empirical evaluation , 1997 .

[53]  A. Renkl,et al.  Instructional Aids to Support a Conceptual Understanding of Multiple Representations. , 2009 .

[54]  Ian M. McDonough,et al.  Learning by analogy: Discriminating between potential analogs , 2010 .

[55]  F. Paas,et al.  Attention guidance in learning from a complex animation: Seeing is understanding? , 2010 .

[56]  Rajesh P. N. Rao,et al.  Embodiment is the foundation, not a level , 1996, Behavioral and Brain Sciences.

[57]  Nuala Ryder,et al.  I hear what you say but I see what you mean: The role of gestures in children's pragmatic comprehension , 2011 .

[58]  S. E. Weismer,et al.  The influence of prosodic and gestural cues on novel word acquisition by children with specific language impairment. , 1993, Journal of speech and hearing research.

[59]  Mark Steedman,et al.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents , 1994, SIGGRAPH.

[60]  R. Mayer,et al.  An embodiment effect in computer-based learning with animated pedagogical agents. , 2012, Journal of experimental psychology. Applied.

[61]  Soyoung Kim,et al.  The Effects of Agent Nonverbal Communication on Procedural and Attitudinal Learning Outcomes , 2008, IVA.

[62]  S. Goldin-Meadow,et al.  From Action to Abstraction , 2014, Psychological science.

[63]  Robert D. Macredie,et al.  Moving face-to-face communication to Web-based systems , 2007, Interact. Comput..

[64]  Eric Jamet,et al.  Attention guiding in multimedia learning , 2008 .

[65]  J. Burgoon,et al.  Nonverbal Communication , 2018, Encyclopedia of Evolutionary Psychological Science.

[66]  Susan Goldin-Meadow,et al.  Hands in the air: using ungrounded iconic gestures to teach children conservation of quantity. , 2008, Developmental psychology.

[67]  Jessica K. Hodgins,et al.  Conversing with children: cartoon and video people elicit similar conversational behaviors , 2014, CHI.

[68]  Robert K. Atkinson,et al.  Using animations and visual cueing to support learning of scientific concepts and processes , 2011, Comput. Educ..

[69]  Susan M. Wagner,et al.  Probing the Mental Representation of Gesture: Is Handwaving Spatial?. , 2004 .

[70]  S. Cook,et al.  Consolidation and transfer of learning after observing hand gesture. , 2013, Child development.