Pantomime Strategies: On Regularities in How People Translate Mental Representations into the Gesture Modality

Pantomime Strategies: On Regularities in How People Translate Mental Representations into the Gesture Modality Karin van Nispen (k.vannispen@tilburguniversity.edu) 1 Tilburg center for Cognition and Communication (TiCC), Tilburg University, The Netherlands Mieke van de Sandt-Koenderman (m.sandt@rijndam.nl) 2 Rotterdam Neurorehabiltation Research (RoNeRes), Erasmus University Rotterdam, The Netherlands Lisette Mol (l.mol@tilburguniversity.edu) 1 Emiel Krahmer (e.j.krahmer@tilburguniversity.edu) 1 Abstract Pantomimes are gestures that occur in absence of speech, which have no conventional meaning. Since their meaning is not conventionalized, the question arises as to what extent they are idiosyncratic. To study this, we collected pantomimes for a standardized set of objects and annotated what representation techniques people used. This resulted in the (to our knowledge) first database of pantomimes. Analyses show that there are regularities in the use of pantomime. That is similar techniques are used for objects across individuals. This shows that pantomime is not fully idiosyncratic. As pantomime is based on people’s mental representation of objects, the observed regularities seem to be a result of intrinsically similar mental representations. Our database gives insight into pantomime 'norms' and could be used as a baseline against which clinical groups (e.g., people with aphasia) can be compared. Keywords: Pantomime; Mental representation; Iconicity; Idiosyncrasy; Non-verbal communication Introduction Pantomimes are hand gestures that occur in absence of speech (McNeill, 2000). They may not be used as frequently as co-speech gestures, but their use can be convenient in situations where speaking is difficult (for instance in a bar, where the music is very loud). The meaning of pantomime is not determined by any convention (McNeill, 2000). That is, the form and meaning of pantomime gestures does not meet any kind of socially constituted group standard (in contrast to emblematic gestures, whose meaning is culturally defined, as for instance for ‘the thumps up’ emblem). Does this mean that the construction of pantomime is idiosyncratic? We know that in the production of co-speech gestures, which are assumed to be non-conventionalized as well, certain similarities between speakers may nevertheless arise. Turkish and Japanese speakers, for instance, represent manner and trajectory in separate gestures more often than English speakers. This is thought to be a result of conventions in spoken language (Kita & Ozyurek, 2003). Language is unlikely to influence pantomime though (Goldin-Meadow, So, Ozyurek, & Mylander, 2008), as it is produced in absence of speech. Does this mean that different people use different pantomimes for the same concepts, or will there be certain regularities? If people cannot rely on linguistic knowledge when producing pantomime, what other sources can they draw from? They may rely on the mental concept of an object 1 , such as a whistle. According to Barsalou (1999), one's mental representation of an object includes perceptually based representations, such as knowledge of the shape, use and sound of a whistle. Although individual experiences with the world (and for this example with ‘whistles’) may differ, for people within a certain culture or community there probably is a great deal of overlap or correspondence between such experiences. This results in similarities across individual’s mental representations. When producing a pantomime, people probably rely on these representations and translate them into pantomime. To this aim, they might use iconicity, which is a similarity between form and meaning (Muller, 1998). For a whistle, iconicity may for instance show in a mapping between the shape of the object (a cylinder with a small extension, see Figure 1) and a hand shape that is similar to this (a fist with slightly stretched index and middle finger). Importantly, not everything is easily represented in pantomime. A first restriction lays in the constraints of the gesture modality. In the gesture domain, one can easily depict physical or spatial properties (e.g. Alibali, 2005), but other properties (for instance color and sound) may be more challenging. As a consequence, for depicting an object in pantomime, people have to select a conceptual feature from their mental representation that meets the constraints of the pantomime domain. Second, for reasons of efficiency people will not express all features that meet these criteria. This leads to the question of how people select the features suitable to depict in pantomime. As McRae, Cree, Seidenberg, and McNorgan (2005) describe in their database of semantic object norms, there may be many features associated with an object (see Table 1 for an overview of features related to a whistle). These features can reflect a variety of basic knowledge types, such as information on its sound, shape and function (based on Wu & Barsalou, 2009). These features can be divided into salient or distinctive and non-distinctive features. In the dataset of McRae et al. (2005), a feature is distinctive when it is not used for any of the other objects. This paper will only focus on objects, animals and plants. For reasons of efficiency we will refer to these categories as ‘objects’.