Focusing Attention with Deictic Gestures and Linguistic Expressions

Focusing Attention with Deictic Gestures and Linguistic Expressions Max M. Louwerse (mlouwers@memphis.edu) Department of Psychology / Institute for Intelligent Systems Memphis, TN 38152 USA Adrian Bangerter (adrian.bangerter@unine.ch) Groupe de Psychologie Appliquee CH - 2000 Neuchâtel, Switzerland Abstract Comprehension and production of text and discourse do not solely depend on linguistic expressions, but also on the physical context. The questions addressed in this study are 1) whether deictic gestures are substitutable for deictic expressions, and 2) whether deictic gestures establish joint attention. An eye tracking experiment investigated the effect of referring expressions and gestures on various aspects of attention. Results indicated that deictic gestures substitute for location descriptions. Furthermore, the manipulation of the synchrony between gesture and speech showed that hearers benefit from focusing of visual attention. Introduction Understanding and production of naturally occurring language does not solely rely on linguistic modalities like content, prosody and text or dialog structure. It also very much relies on non-linguistic modalities like eye gaze, facial expressions, body posture and gestures. It seems that gestures fulfill an important supportive role in bringing about the communicative project: everybody uses them, whether they are pointing out directions on the map, emphasizing a point they are trying to make, whether they are in a face-to-face argument or chatting on a cell phone. So why do we gesture? There are several explanations that are not necessarily mutually exclusive. According to one account, we gesture to facilitate lexical access (Butterworth & Beattie, 1978; Rime & Schiaratura, 1991). The timing gap between a gesture and an unfamiliar word is larger than between a gesture and a familiar word (Morrel-Samuels & Krauss, 1992). Furthermore, gesture is associated with fluent speech: when the speech is disrupted, like in stuttering, the gesture is halted (Mayberry & Jaques, 2000). According to a second account, gestures facilitate thinking (Goldin-Meadow, 2003; McNeill, 1992). Gesture and speech are coexpressive manifestations of one integrated system. They form complementary components of one underlying process and thereby help organizing thought. Indeed, children’s performance on counting tasks improves when they gesture (Alibali & DiRusso, 1999). According to both of these accounts, gestures help speakers but not necessarily hearers. By a third account, gestures support communicative joint activities, that is, they are informative for hearers (Clark, 1996). The speaker and hearer are participating in the joint project of communication. Gesture is thereby part of the language use. Evidence for this account comes for instance from Ozyurek (2002) who showed that speakers change orientation of their gestures dependent on their hearers. However, these three different accounts investigate the production of gestures. An important question that remains is how addressees perceive these gestures. According to the first two accounts the effect gestures have on the addressee is irrelevant, according to the third account there is an immediate impact. There is strong evidence that gesture is intrinsically related to (symbolic) language processing (Butterworth and Morrisette, 1996). For instance, 90% of all gestures occur when we speak (McNeill, 1992). Furthermore, there are close ties between gesture and language development (Butcher & Goldin-Meadow, 2000). Also, humans are the only species that gesture (Butterworth, 2003; Povinelli, Bering, Giambrone, 2003). At the same time gestures are very different than linguistic cues. For instance, despite the fact that interlocutors rely on cues from gestures, particularly when the speech is ambiguous (Thompson & Massaro, 1996) or when the environment is noisy (Rogers, 1978), they are often unable to remember what hand gestures they have seen (Krauss, Morrell-Samuels, et al. 1991). Gestures thus seem to fulfill an important but subtle function: They have close ties to the linguistic system and seem to be intrinsically integrated with it, at the same time there are differences in production and understanding. An important research question is therefore what the relation is between gestures and linguistic expressions. It is important to keep in mind that we have a large range of gestures available. Kendon (1988) nicely places hand gestures along a continuum from 1) gesticulation, 2) language-like gestures and 3) emblems to 4) sign languages. Moving from left to right along the continuum gestures are replacing the role of speech; hardly so in gesticulation, very much so in sign language. This paper uses gesture to solely refer to gesticulation. Within gestures different types can be identified (Ekman & Friesen, 1969; Goldin-Meadow, 2003; McNeill, 1992). Generally, four categories are distinguished: 1) iconic gestures that mimic the object being represented through the gesture (making sawing movements when talking about sawing a tree), 2) concrete deictic gestures (pointing at a painting when talking about the Rembrandt’s Nightwatch), 3) abstract deictic gestures (gesturing from left to right saying “from the beginning to the end”); 4) beat movements (used in the rhythm of the speech or to mark important intonational boundaries). In this paper we will focus on concrete deictic gestures