Gesture structure affects syntactic structure in speech

Gesture Structure Affects Syntactic Structure in Speech Lisette Mol (l.mol@uvt.nl) Tilburg center for Cognition and Communication (TiCC), School of Humanities, Tilburg University P.O. Box 90135, NL-5000 LE Tilburg, The Netherlands Sotaro Kita (s.kita@bham.ac.uk) University of Birmingham, School of Psychology, Birmingham B15 2TT, UK Abstract and linguistic processing. Thus, in this view, gesture and speech are two outcomes of a single process. In addition to the line of thought that gestures are intended communicatively, it has also been proposed that there are speaker-internal motivations for gesture production. Some propose that gesture production facilitates cognitive processes in general, by lightening cognitive load (Goldin- Meadow, Nusbaum, Kelly & Wagner, 2001). Others propose that gesture production facilitates a specific process in speech production. In this article, we focus on the latter class of proposals. There are three prominent proposals in the literature: The Lexical Retrieval Hypothesis, the Image Activation Hypothesis, and the Information Packaging Hypothesis. The Lexical Retrieval Hypothesis (LRH) states that gestures facilitate the retrieval of lexical items from the mental lexicon (Krauss, Chen, & Gottesman, 2000). In this view, gesture production is based on spatial imagery in working memory. Rather than there being an interplay between the processes of gesture and speech production, the execution of a gesture is thought to activate spatio-dynamic features, which in turn activate conceptual information. Through cross-modal priming, this aids the retrieval of lexical items. Thus, gesture production precedes speech formulation entirely. The Image Activation Hypothesis (IAH) states that gesturing serves to keep an image (Freedman, 1977) or certain spatial features (De Ruiter, 1998) activated while they are encoded by the process of speech formulation. The Information Packaging Hypothesis (Kita, 2000) critically differs from the Lexical Retrieval Hypothesis and the Image Activation Hypothesis in its assumptions on the for-speaker motivations of gesture production, and on the interplay between gesture and speech production. Rather than simply activating information or maintaining the activation of spatial information, gesture production is thought to structure information, and to package it into units that are suitable for the speech formulation process. Like Growth Point Theory, the Information Packaging Hypothesis (IPH) assumes that different forms of processing underlie gesture and speech. It is proposed that gesture is based on spatio-motoric processing and speech on analytic processing. The IPH assumes that [s]patio-motoric thinking, which underlies representational gestures, helps speaking by providing an alternative informational Different functions have been proposed for the hand gestures speakers spontaneously produce while speaking. The Information Packaging Hypothesis (Kita, 2000) states that gestures can structure rich spatio-motoric information into packages suitable for speaking. It therefore predicts that how information is divided over different gestures affects how it is divided over different processing units in speech: clauses. We indeed found that if participants were asked to express the manner and path of a motion in one gesture, they were also more likely to conflate this information into one clause in speech, whereas if they were asked to produce separate gestures, they were more likely to express manner and path in separate clauses too. These results support the view that there are speaker-internal motivations for gesture production. They confirm predictions made by the Information Packaging Hypothesis, which the Lexical Retrieval Hypothesis and the Image Activation Hypothesis do not make. Keywords: Gesture; Speech; Production; Motion Event Introduction When speaking, most people tend to produce hand gestures that are closely synchronized with their speech semantically (e.g. McNeill, 2005), temporally (e.g. Chui, 2005), and structurally (e.g. Kita & Ozyurek, 2003). Because of this careful coordination, it is generally assumed that the processes of speech and gesture production are somehow related. Yet what is the exact role of gesture production in relation to speech production? Gesture and Speech Production In this paper, we focus on representational hand gestures (McNeill, 1992). Representational gestures either depict action, motion or shape ( iconic gestures ) or indicate a location or direction ( deictic gesture ). Much evidence has been gathered in support of a theory that (representational) gestures, like speech, are part of a speaker's communicative effort (Kendon, 2004). In line with this view, Growth Point Theory (McNeill, 2005; McNeill & Duncan, 2010) starts from the observation that gesture and speech co-express idea units, each using a different form of semiosis. While gesture employs a global/synthetic form of representation, speech is expressed in an analytic/ combinatoric form. It is assumed that gesture and speech production share a common origin: the growth point. From this origin, a bimodal utterance develops from the interplay of imagistic