Modality Choice for Generation of Referring Acts: Pointing versus Describing

The main aim of this paper is to challenge two commonly held assumptions regarding modality selection in the generation of referring acts: the assumption that non-verbal means of referring are secondary to verbal ones, and the assumption that there is a single strategy that speakers follow for generating referring acts. Our evidence is drawn from a corpus of task-oriented dialogues that was obtained through an observational study. We propose two alternative strategies for modality selection based on correlation data from the observational study. Speakers that follow the first strategy simply abstain from pointing. Speakers that follow the other strategy make the decision whether to point dependent on whether the intended referent is in focus and/or important. This decision precedes the selection of verbal means (i.e., words) for referring.

[1]  G. Butterworth.  Pointing Is the Royal Road to Language for Babies , 2003 .

[2]  Ahm Cremers,et al.  Dutch and English demonstratives : a comparison , 1996 .

[3]  Mitsuru Ishizuka,et al.  Life-like characters - tools, affective functions, and applications , 2004, Life-like characters.

[4]  A. Bangerter,et al.  Using Pointing and Describing to Achieve Joint Focus of Attention in Dialogue , 2004, Psychological science.

[5]  Harry Bunt,et al.  Multimodal Human-Computer Communication , 1995, Lecture Notes in Computer Science.

[6]  Siobhan Chapman.  Logic and Conversation , 2005 .

[7]  Vincent J. van Heuven,et al.  The significance of demonstrative position in modern Dutch , 1988 .

[8]  S. Kita.  Pointing: Where language, culture, and cognition meet , 2003 .

[9]  Wim Claassen,et al.  Generating Referring Expressions in a Multimodal Environment , 1992, NLG.

[10]  Maddy D. Brouwer-Janse,et al.  Human-Machine Communication for Educational Systems Design , 1994 .

[11]  H. H. Clark,et al.  Changing Ideas about Reference , 2004 .

[12]  Justin Buchler.  The Philosophy of Peirce: Selected Writings , 1941 .

[13]  Anita Cremers,et al.  Reference to objects : an empirically based study of task-oriented dialogues , 1996 .

[14]  A. Kendon,et al.  Pointing by Hand in “Neapolitan” , 2003 .

[15]  Emiel Krahmer,et al.  A New Model for Generating Multimodal Referring Expressions , 2003, ENLG@EACL.

[16]  Darren Gergle.  What's There to Talk About? A Multi-Modal Model of Referring Behavior in the Presence of Shared Visual Information , 2006, EACL.

[17]  Joseph Bates,et al.  The role of emotion in believable agents , 1994, CACM.

[18]  Thies Pfeiffer,et al.  Measuring and Reconstructing Pointing in Visual Contexts , 2006 .

[19]  James C. Lester,et al.  Deictic Believability: Coordinated Gesture, Locomotion, and Speech in Lifelike Pedagogical Agents , 1999, Appl. Artif. Intell..

[20]  Helen Blood,et al.  Body Language and the Social Order: Communication as Behavioral Control , 1974 .

[21]  Carla Huls,et al.  Studies into Full Integration of Language and Action , 1995, Multimodal Human-Computer Communication.

[22]  Kees van Deemter,et al.  The Clarity-Brevity Trade-off in Generating Referring Expressions , 2006, INLG.

[23]  Harry Bunt,et al.  Multimodal Human-Computer Communication: Systems Techniques, and Experiments. Lecture Notes in Artificial Intelligence , 1998 .

[24]  Robbert-Jan Beun,et al.  Object reference in a shared domain of conversation , 1998 .

[25]  Emiel Krahmer,et al.  Generating Referring Expressions in a Multimodal Context An empirically oriented approach , 2000, CLIN.

[26]  C. Y. Thielman,et al.  Natural Language with Integrated Deictic and Graphic Gestures , 1989, HLT.

[27]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[28]  Charles S. Peirce,et al.  The Philosophy of Peirce : Selected Writings , 1941 .

[29]  Leo G. M. Noordman,et al.  Demonstrative nominal anaphors: a case of nonidentificational markedness , 1995 .

[30]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[31]  Ipke Wachsmuth,et al.  Incremental Generation of Multimodal Deixis Referring to Objects , 2005, ENLG.

[32]  Emiel Krahmer,et al.  Proceedings of the Workshop on Multimodal Output Generation (MOG 2007) , 2007 .

[33]  James C. Lester,et al.  The persona effect: affective impact of animated pedagogical agents , 1997, CHI.

[34]  Barbara J. Grosz,et al.  The representation and use of focus in dialogue understanding. , 1977 .

[35]  Brouwer-Janse,et al.  Referring in a shared workspace , 1994 .