Facilitating Understanding of Movements in Dynamic Visualizations: an Embodied Perspective

Learners studying mechanical or technical processes via dynamic visualizations often fail to build an accurate mental representation of the system’s movements. Based on embodied theories of cognition assuming that action, perception, and cognition are closely intertwined, this paper proposes that the learning effectiveness of dynamic visualizations could be enhanced by grounding the movements of the presentation in people’s own bodily experiences during learning. We discuss recent research on embodied cognition and provide specific strategies for how the body can be used to ground movements during the learning process: (1) making or observing gestures, (2) manipulating and interacting with objects, (3) using body metaphors, and (4) using eye movements as retrieval cues. Implications for the design of dynamic visualizations as well as directions for future research are presented.

[1]  Scotty D. Craig,et al.  Animated Pedagogical Agents in Multimedia Educational Environments: Effects of Agent Properties, Picture Features, and Redundancy , 2002 .

[2]  J. S. Nairne.  The myth of the encoding-retrieval match , 2002, Memory.

[3]  N. Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[4]  R. Passingham,et al.  Action observation and acquired motor skills: an FMRI study with expert dancers. , 2005, Cerebral cortex.

[5]  Barbara Tversky,et al.  Animation: can it facilitate? , 2002, Int. J. Hum. Comput. Stud..

[6]  L. Barsalou,et al.  Situating Abstract Concepts , 2004 .

[7]  F. Paas,et al.  Attention guidance in learning from a complex animation: Seeing is understanding? , 2010 .

[8]  Daniel L. Schwartz,et al.  Shuttling Between Depictive Models and Abstract Rules: Induction and Fallback , 1996, Cogn. Sci..

[9]  I. Eibl-Eibesfeldt.  Human ethology: concepts and implications for the sciences of man , 1979 .

[10]  Richard Lowe,et al.  Interrogation of a dynamic visualization during learning , 2004 .

[11]  D. Schwartz,et al.  Tool use and the effect of action on the imagination. , 2000, Journal of experimental psychology. Learning, memory, and cognition.

[12]  R. Johansson,et al.  Action plans used in action observation , 2003, Nature.

[13]  R. Klatzky.  The role of motor representations in semantic sensibility judgments , 1989 .

[14]  Daniel C. Richardson,et al.  Movements of eye and hand in language and cognition. , 2008 .

[15]  S. Goldin-Meadow,et al.  Making children gesture brings out implicit knowledge and leads to learning. , 2007, Journal of experimental psychology. General.

[16]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[17]  M A Srinivasan,et al.  Development and evaluation of an epidural injection simulator with force feedback for medical training. , 2001, Studies in health technology and informatics.

[18]  Michael J. Spivey,et al.  Graded motor responses in the time course of categorizing atypical exemplars , 2007, Memory & cognition.

[19]  J. Piaget,et al.  The Psychology of the Child , 1969 .

[20]  G. Lakoff,et al.  Metaphors We Live By , 1980 .

[21]  Robert K. Atkinson,et al.  Animated pedagogical agents: does their degree of embodiment impact learning from static or animated worked examples? , 2007 .

[22]  Rolf A. Zwaan,et al.  The Effect of Implied Orientation Derived from Verbal Context on Picture Recognition , 2001, Psychological science.

[23]  Susan M. Wagner,et al.  Explaining Math: Gesturing Lightens the Load , 2001, Psychological science.

[24]  Raymond W. Gibbs,et al.  Real and Imagined Body Movement Primes Metaphor Comprehension , 2007, Cogn. Sci..

[25]  A. Glenberg,et al.  Processing Abstract Language Modulates Motor System Activity , 2008, Quarterly journal of experimental psychology.

[26]  G. Olympiou,et al.  Physical versus Virtual Manipulative Experimentation in Physics Learning. , 2011 .

[27]  Hanne Pedersen,et al.  Learning with animation , 2011 .

[28]  G. Lakoff,et al.  The Brain's concepts: the role of the Sensory-motor system in conceptual knowledge , 2005, Cognitive neuropsychology.

[29]  Alejandro Lleras,et al.  Swinging into thought: Directed movement guides insight in problem solving , 2009, Psychonomic bulletin & review.

[30]  Arthur M. Glenberg,et al.  Indexical understanding of instructions , 1999 .

[31]  Lena A. E. Tibell,et al.  Do haptic representations help complex molecular learning , 2011 .

[32]  G. Rizzolatti,et al.  Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study , 2001, The European journal of neuroscience.

[33]  Richard K. Lowe,et al.  A unified view of learning from animated and static graphics , 2008 .

[34]  Fred Paas,et al.  Making instructional animations more effective: a cognitive load approach , 2007 .

[35]  I. Johnsrude,et al.  Somatotopic Representation of Action Words in Human Motor and Premotor Cortex , 2004, Neuron.

[36]  Christine D. Wilson,et al.  Grounding conceptual knowledge in modality-specific systems , 2003, Trends in Cognitive Sciences.

[37]  Michael P. Kaschak,et al.  Body posture facilitates retrieval of autobiographical memories , 2007, Cognition.

[38]  J. Bruner.  Toward a Theory of Instruction , 1966 .

[39]  S. Schwan,et al.  The cognitive benefits of interactive videos: learning to tie nautical knots , 2004 .

[40]  Michael P. Kaschak,et al.  Grounding language in action , 2002, Psychonomic bulletin & review.

[41]  F. Paas,et al.  Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting. , 2005, Journal of experimental psychology. Applied.

[42]  Lara M. Triona,et al.  Hands on What? The Relative Effectiveness of Physical versus Virtual Materials in an Engineering Design Project by Middle School Children , 2007 .

[43]  R. Ilmoniemi,et al.  Functional links between motor and language systems , 2005, The European journal of neuroscience.

[44]  R. Atkinson.  Optimizing learning from examples using animated pedagogical agents. , 2002 .

[45]  Lara M. Triona,et al.  Point and Click or Grab and Heft: Comparing the Influence of Physical and Virtual Instructional Materials on Elementary School Students' Ability to Design Experiments , 2003 .

[46]  R. Mayer,et al.  When learning is just a click away: Does simple user interaction foster deeper understanding of multimedia messages? , 2001 .

[47]  Daniel C. Richardson,et al.  When facts go down the rabbit hole: Contrasting features and objecthood as indexes to memory , 2008, Cognition.

[48]  Shahrzad Mahootian,et al.  The Role of Gesture in Bilingual Education: Does Gesture Enhance Learning? , 2004 .

[49]  Zenzi M. Griffin,et al.  Why Look? Reasons for Eye Movements Related to Language Production. , 2004 .

[50]  S. Kelly,et al.  Broadening the units of analysis in communication: speech and nonverbal behaviours in pragmatic comprehension , 2001, Journal of Child Language.

[51]  Susan Goldin-Meadow,et al.  Gesturing Saves Cognitive Resources When Talking About Nonpresent Objects , 2010, Cogn. Sci..

[52]  Steve Benford,et al.  The role of physical representations in solving number problems: A comparison of young children's use of physical and virtual materials , 2010, Comput. Educ..

[53]  Mireille Betrancourt,et al.  The Cambridge Handbook of Multimedia Learning: The Animation and Interactivity Principles in Multimedia Learning , 2005 .

[54]  Mary Hegarty,et al.  The Role of Gestures in Mental Animation , 2005, Spatial Cogn. Comput..

[55]  S. Goldin-Meadow,et al.  The Role of Gesture in Learning: Do Children Use Their Hands to Change Their Minds? , 2006 .

[56]  David N. Rapp,et al.  The ‘Ins’ and ‘Outs’ of Learning: Internal Representations and External Visualizations , 2008 .

[57]  L. Barsalou.  Grounded cognition. , 2008, Annual review of psychology.

[58]  Rolf A. Zwaan,et al.  Seeing, acting, understanding: motor resonance in language comprehension. , 2006, Journal of experimental psychology. General.

[59]  R. Passingham,et al.  Seeing or Doing? Influence of Visual and Motor Familiarity in Action Observation , 2006, Current Biology.

[60]  Lei Mo,et al.  Dynamic Mental Representations in Language Comprehension , 2009, Perceptual and motor skills.

[61]  Richard E. Clark,et al.  Cognitive and Affective Benefits of an Animated Pedagogical Agent for Learning English as a Second Language , 2006 .

[62]  Daniel C. Richardson,et al.  Representation, space and Hollywood Squares: looking at things that aren't there anymore , 2000, Cognition.

[63]  Janette Atkinson,et al.  The Developing Visual Brain , 2000 .

[64]  M. Hegarty,et al.  Learning With Real Machines or Diagrams: Application of Knowledge to Real-World Problems , 1995 .

[65]  S. Goldin-Meadow,et al.  Gesturing makes learning last , 2008, Cognition.

[66]  Bruno Laeng,et al.  Eye scanpaths during visual imagery reenact those of perception of the same visual scene , 2002, Cogn. Sci..

[67]  M. Wexler,et al.  Movement prediction and movement production. , 2001, Journal of experimental psychology. Human perception and performance.

[68]  J. Henderson,et al.  Taking a new look at looking at nothing , 2008, Trends in Cognitive Sciences.

[69]  Michael J. Spivey,et al.  Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects , 2001, Psychological research.

[70]  Karen R. Brandt,et al.  Reinstating effortful encoding operations at test enhances episodic remembering , 2007, Quarterly journal of experimental psychology.

[71]  G. Lakoff,et al.  Philosophy in the flesh : the embodied mind and its challenge to Western thought , 1999 .

[72]  Mary Hegarty,et al.  The Roles of Mental Animations and External Animations in Understanding Mechanical Systems , 2003 .

[73]  Soyoung Kim,et al.  Designing nonverbal communication for pedagogical agents: When less is more , 2009, Comput. Hum. Behav..

[74]  Rolf A. Zwaan,et al.  Moving words: dynamic representations in language comprehension , 2004, Cognitive Sciences.

[75]  John Sweller,et al.  The Mirror Neuron System and Observational Learning: Implications for the Effectiveness of Dynamic Visualizations , 2009 .

[76]  Susan Goldin-Meadow,et al.  Hands in the air: using ungrounded iconic gestures to teach children conservation of quantity. , 2008, Developmental psychology.

[77]  G. Altmann,et al.  The Oxford Handbook of Psycholinguistics , 2007 .

[78]  S. Goldin-Meadow,et al.  Hearing Gesture: How Our Hands Help Us Think , 2003 .

[79]  P Haggard,et al.  The psychology of action. , 2001, British journal of psychology.

[80]  J. Decety,et al.  The power of simulation: Imagining one's own and other's behavior , 2006, Brain Research.

[81]  Eric Jamet,et al.  Using video and static pictures to improve learning of procedural contents , 2009, Comput. Hum. Behav..

[82]  Arthur M. Glenberg,et al.  Improving Native American children's listening comprehension through concrete representations. , 2007 .

[83]  Béatrice S. Hasler,et al.  Learner Control, Cognitive Load and Instructional Animation , 2007 .

[84]  David N. Rapp,et al.  The ‘ins’ and ‘outs’ of learning: Internal representations and external visualizations: Theory and Practice in Science Education , 2008 .

[85]  A. Koriat,et al.  Memory organization of action events and its relationship to memory performance. , 2003, Journal of experimental psychology. General.

[86]  Brice Isableu,et al.  Embodied spatial transformations: "body analogy" for the mental rotation of objects. , 2006, Journal of experimental psychology. General.

[87]  Paul Ayres,et al.  Learning hand manipulative tasks: When instructional animations are superior to equivalent static representations , 2009, Comput. Hum. Behav..

[88]  Richard Mayer,et al.  Multimedia Learning , 2001, Visible Learning Guide to Student Achievement.

[89]  D. Leutner,et al.  Instructional animation versus static pictures: A meta-analysis , 2007 .

[90]  Susan Goldin-Meadow,et al.  Children Learn When Their Teacher's Gestures and Speech Differ , 2005, Psychological science.

[91]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[92]  Lawrence W. Barsalou,et al.  Grounded Cognition: Past, Present, and Future , 2010, Top. Cogn. Sci..

[93]  A. Glenberg.  Language and action: creating sensible combinations of ideas , 2007 .

[94]  A. Glenberg,et al.  Symbols and Embodiment: Debates on Meaning and Cognition , 2008 .

[95]  G. A. Miller.  The magical number seven plus or minus two: some limits on our capacity for processing information. , 1956, Psychological review.

[96]  Daniel C. Richardson,et al.  Multimodal events and moving locations: eye movements of adults and 6-month-olds reveal dynamic spatial indexing. , 2004, Journal of experimental psychology. General.

[97]  Michael P. Kaschak,et al.  Short article: Encoding in verbal, enacted and autobiographical tasks in young and older adults , 2006, Quarterly journal of experimental psychology.

[98]  Michael P. Kaschak,et al.  Activity and Imagined Activity Can Enhance Young Children's Reading Comprehension. , 2004 .

[99]  Yuki Kamide,et al.  Now you see it, now you don't: mediating the mapping between language and the visual world , 2004 .

[100]  Fred Paas,et al.  Learning by generating vs. receiving instructional explanations: Two approaches to enhance attention cueing in animations , 2010, Comput. Educ..

[101]  Lawrence W. Barsalou,et al.  Representing Properties Locally , 2001, Cognitive Psychology.

[102]  G. Rizzolatti,et al.  The mirror-neuron system. , 2004, Annual review of neuroscience.

[103]  L. Barsalou,et al.  Whither structured representation? , 1999, Behavioral and Brain Sciences.

[104]  A. Glenberg,et al.  Averting the gaze disengages the environment and facilitates remembering , 1998, Memory & cognition.

[105]  A. Glenberg,et al.  What memory is for: Creating meaning in the service of action , 1997, Behavioral and Brain Sciences.

[106]  Miriam Reiner,et al.  Visualization : theory and practice in science education , 2008 .

[107]  Vittorio Gallese,et al.  Listening to Action-related Sentences Activates Fronto-parietal Motor Circuits , 2005, Journal of Cognitive Neuroscience.

[108]  Jeffrey D. Karpicke,et al.  The Power of Testing Memory Basic Research and Implications for Educational Practice , 2006 .

[109]  David H. Jonassen,et al.  Handbook of Research for educational Communications and Technology , 1997 .

[110]  R. Klatzky,et al.  Teachers’ gestures facilitate students’ learning: A lesson in symmetry , 2003 .

[111]  Daniel R. Montello,et al.  Spatial Reasoning With External Visualizations: What Matters Is What You See, Not Whether You Interact , 2008, Cogn. Sci..

[112]  Rolf A. Zwaan.  Embodied cognition, perceptual symbols, and situation models , 1999 .

[113]  Michael P. Kaschak,et al.  Putting words in perspective , 2004, Memory & cognition.

[114]  Xiaojin Zhu,et al.  Improving early reading comprehension using embodied CAI , 2011 .

[115]  Rolf A. Zwaan,et al.  Language Comprehenders Mentally Represent the Shapes of Objects , 2002, Psychological science.

[116]  John Sweller,et al.  Instructional animations can be superior to statics when learning human motor skills , 2009, Comput. Hum. Behav..

[117]  Mary Hegarty,et al.  Individual differences in use of external visualisations to perform an internal visualisation task. , 2007 .