From Hands to Minds: Gestures Promote Action Understanding

From Hands to Minds: Gestures Promote Action Understanding Seokmin Kang (sk2587@columbia.edu) Teachers College, Columbia University New York, NY 10027 USA Barbara Tversky (btversky@stanford.edu) Columbia Teachers College New York, NY 10027 USA Stanford University Stanford, CA 94305 USA John B. Black (jbb21@columbia.edu) Teachers College, Columbia University New York, NY 10027 USA point to places or things in the world or in a virtual world. Iconic gestures show what something looks like or acts like (e.g., McNeill, 1992; Goldin-Meadow, 2003). Together, these kinds of gestures can carry rich semantic content. A train of integrated deictic and iconic gestures can be used on virtual stages to create detailed models of situations, such as environments (e. g., Emmorey, Tversky, & Taylor, 2000) and actions, such as how a lock works (e. g., Engle, 1998). Are such gestures successful in communicating knowledge as well as in representing it? Abstract Understanding dynamic concepts is more difficult than understanding static ones. The present study showed that understanding dynamic concepts can be enhanced by gestures that convey action. Participants learned how an engine worked from one of two videos, with identical verbal scripts and identical diagrams. One video was accompanied by gestures showing the structure of the system; the other was accompanied by gestures showing the actions of the system. Both groups learned the basics of the system. Participants who saw the action gestures depicted more dynamic information in their visual explanations of the system and included more dynamic information in their verbal explanations of the system. Because they are inherently dynamic, gestures appear to be especially suited for conveying dynamic information. Knowledge on the page As such, sequences of organized gestures can serve much like diagrams. In fact, many kinds of gestures can be mapped to kinds of diagrammatic features; that is, they carry the same meanings (Tversky, Heiser, Lee, & Daniel, 2009). Diagrams have some advantages over gestures as a means of representing knowledge. Diagrams have permanence, so they can be inspected and reinspected. Because they are external and persist, they do not need to be kept in mind, so the mind is free to use the diagram as a basis for reorganization, for inference, and for discovery. Diagrams use elements and spatial relations on a page to represent elements and relations that are actually spatial, as in maps or architectural plans, or that are metaphorically spatial, as in the periodic table or organization charts (e. g., Tversky, 2011; Tversky, et al., 2009). Gestures, like language, are external, but lack permanence. A series of gestures used to create a model of a situation requires working memory to create, understand, and remember, and can tax working memory. On the other hand, diagrams are static, so it can be challenging to convey action, change, and process in diagrams. Typically, arrows are used, but they can be ambiguous (e. g., Heiser & Tversky 2006; Tversky, 2011; Tversky, Heiser, MacKenzie, Lozano, & Morrison, 2007). Gestures are by nature dynamic, so they can portray action, if schematically (e. g., Kita & Ozyurek, 2003; Rizzolatti & Arbib, 1998). In fact, when gestures are used with diagrams in explanations, diagrams are often used to convey structure, and gestures to portray action (e. g., Engle, 1998). Keywords: gesture; diagram; complex systems; knowledge construction Knowledge in the hands When people explain something, they typically use gestures as well as speech. Gestures can carry information that is redundant with speech, reinforcing the message by presenting information in two modalities. Importantly, gestures sometimes carry information that is not carried in speech (e. g, Bavelas, 1994; Church & Goldin-Meadow, 1986; Perry, Church, & Goldin-Meadow, 1988). In some cases, speech refers listeners to gesture, as in “turn this way,” but in other cases, there is no cuing of the gestures. Nevertheless, the information carried solely in gesture can reveal the thought of speakers and affect the thought of both those who make gestures and those who watch them (e.g., Beattie & Shovelton, 1999; Chu & Kita, 2011; Goldin- Meadow, Cook, & Mitchell, 2009; Goldin-Meadow, Kim, & Singer, 1999; Hegarty, Mayer, Kriz, & Keehner, 2005; Kessell & Tversky, 2006; Singer & Goldin-Meadow, 2005; Mcgregor, Rohlfing, Bean, & Marschner, 2009; Ping & Goldin-Meadow, 2008; Schwartz & Black. 1996; Thompson, Driscoll, & Markson, 1998; Valenzeno, Alibali, & Klatzky, 2003). It is primarily iconic and deictic gestures that reveal the thought of those who make them and affect the thought of those who make them or observe them. Deictic gestures