Talking Through Graphics: An Empirical Study of the Sequential Integration of Modalities

Talking Through Graphics: An Empirical Study of the Sequential Integration of Modalities Ichiro Umata (umata@mic.atr.co.jp) 1 ATR Media Integration & Communications Research Laboratories; Seika Soraku Kyoto, 619-0288 Japan Atsushi Shimojima (ashimoji@jaist.ac.jp) Japan Advanced Institute of Science and Technology; 1–1 Asahi Tatsunokuchi Nomi Ishikawa 923-1292 Japan Yasuhiro Katagiri (katagiri@mic.atr.co.jp) ATR Media Integration & Communications Research Laboratories; Seika Soraku Kyoto, 619-0288 Japan Abstract An empirical investigation was conducted on the character- istics of language use in graphics communication settings. Graphics communications, such as dialogues using maps, drawings, or pictures, provide people with two independent systems of representation, spoken language and graphics. Drawing on our dialogue data, we will show that the presence of a graphical representation significantly changes the way the spoken language is used, extending its expressive capacity in most cases. As two remarkable uses of language affected in this way, we will report the phenomena of mediated reference and dual description, illustrating them with actual examples from our data. Finally, a quantitative analysis of our data will show that these special uses of language are indeed as common as conventional uses of language in the presence of graphical representations. Introduction Conversational exchanges that involve external graphical rep- resentations are quite common in our daily lives. People often give and ask directions by referring to maps, or they might draw a floor plan in discussing where to place furniture in a living room. Now, linguistic expressions denote objects and relations in the world. This denotation relation is governed by conventions inside the language. An utterance of a lin- guistic expression carries unique information about the world through these conventions. This is what the standard view of the semantics of language tells us. However, when we look at speech in conversational exchanges involving graphics, re- gardless of the language used, we will immediately notice utterances that do not conform to this standard picture. This paper is a detailed examination of the impact of graph- ics on the use of language. Our data consist of actual two- party dialogues where participants draw or consult a map dur- ing verbal exchanges. We will focus on two remarkable uses of language, called “mediated reference” and “dual descrip- tion,” that we found through an examination of our data. Both phenomena are clearly specific to dialogues involving some graphical representation, or at least, some external represen- tation other than speech. Briefly, mediated reference is a case where a linguistic ex- pression reaches its “final” referent due to the fact that its “immediate” referent has a referential connection to this final one in the system of graphics. For example, our subjects often use the indexical “kore” (this) to refer to a building or some other landmark, although its immediate referent is clearly an icon on the map; the icon refers to the landmark in the system Also with Kobe University. of map, and this fact somehow enables the indexical expres- sion to do so too. We will discuss more examples of mediated reference later, and introduce three more varieties of the phe- nomenon. Dual description is a case where a declarative sentence is used to describe a fact that holds in the graphic as well as the corresponding fact in the situation represented by the graphic. Suppose, when asked about the number of stations between two particular stations, one counts the number of icons on a railroad map and says, “There are three stations in-between them.” Is this report concerned with the map itself, or with the mapped railroad? Is it reporting that the railroad map has three station icons between two particular station icons, or that the railroad system has three stations between the two stations? Whichever the answer may be, it seems clear that the speaker has managed to describe both facts with this sen- tence. Note that, on the semantics associated with the railroad map, the first fact means the second fact, and this semantic relation somehow underwrites the duplicative use of the sen- tence. Both uses of language are so natural and common in a di- alogue involving a graphical representation that people may not even be aware of the phenomena. In fact, their frequent occurrence in such settings suggests that they are not a de- viant but rather a perfectly legitimate use of language. Yet the empirical research on the integration of linguistic and graph- ical representations has focused on the issue of how speech is used to disambiguate a graphic (Neilson and Lee 1994) or how a graphic is used to disambiguate speech (Lee and Zeevat 1990). The linguistic-graphic integration has been also stud- ied from a logical point of view, but the focus has been on how a graphic expresses what cannot be easily expressed by a linguistic representation (Barwise and Etchemendy 1996, Shimojima 1999). For both views, the fundamental form of linguistic-graphic integration is a parallel one, where each mode of representation expresses information in its own way, but since one mode of representation expresses what the other form does not, they may work complementarily to each other. In contrast, the two phenomena that we are highlighting in this paper point to a rather different form of integration, where the presence of one mode of representation extends the expressive capacity of the other by affecting the way it is used. Our goal is to draw due attention to this sequential form of graphic-linguistic integration by demonstrating that the in- stances of that type of integration are commonly observed in actual human dialogues, as opposed to mere logical possibil- ities. In the next section, we will describe the methods through