Deictic object reference in task-oriented dialogue

This chapter presents an original approach towards a detailed understanding of the usage of pointing gestures accompanying referring expressions. This effort is undertaken in the context of human-machine interaction integrating empirical studies, theory of grammar and logics, and simulation techniques. In particular, we take steps to classify the role of pointing in deictic expressions and to model the focussed area of pointing gestures, the so-called pointing cone. This pointing cone serves as a central concept in a formal account of multi-modal integration at the linguistic speech-gesture interface as well as in a computational model of processing multi-modal deictic expressions.

[1]  Alex Lascarides,et al.  Logics of Conversation , 2005, Studies in natural language processing.

[2]  Marc Erich Latoschik.  Designing transition networks for multimodal VR-interactions using a markup language , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[3]  James C. Lester,et al.  Deictic Believability: Coordinated Gesture, Locomotion, and Speech in Lifelike Pedagogical Agents , 1999, Appl. Artif. Intell..

[4]  Candace L. Sidner,et al.  Attention, Intentions, and the Structure of Discourse , 1986, CL.

[5]  Gennaro Chierchia,et al.  Meaning and Grammar: An Introduction to Semantics , 1990 .

[6]  Norbert Reithinger,et al.  The Performance of an Incremental Generation Component for Multi-Modal Dialog Contributions , 1992, NLG.

[7]  H. Rieser,et al.  Statistical Support for the Study of Structures in Multi-Modal Dialogue : Inter-Rater Agreement and Synchronization , 2004 .

[8]  Ulrike Gut,et al.  The TASX-environment : an XML-based corpus database for time aligned language data , 2001 .

[9]  Daniel Vanderveken,et al.  Foundations of Illocutionary Logic , 1985 .

[10]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[11]  Ipke Wachsmuth,et al.  Incremental Generation of Multimodal Deixis Referring to Objects , 2005, ENLG.

[12]  A. Kendon,et al.  Nonverbal Communication, Interaction, and Gesture , 1981 .

[13]  Ipke Wachsmuth,et al.  Deixis: How to Determine Demonstrated Objects Using a Pointing Cone , 2005, Gesture Workshop.

[14]  Marc Erich Latoschik.  A General Framework for Multimodal Interaction in Virtual Reality Systems: PrOSA , 2001 .

[15]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[16]  J. D. Ruiter.  The production of gesture and speech , 2000 .

[17]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[18]  William C. Mann,et al.  Rhetorical Structure Theory: A Framework for the Analysis of Texts , 1987 .

[19]  W. Mann.  Dialogue games: Conventions of human interaction , 1988 .

[20]  Wim Claassen,et al.  Generating Referring Expressions in a Multimodal Environment , 1992, NLG.

[21]  David McNeill,et al.  Pointing and Morality in Chicago , 1998 .

[22]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[23]  Paul Piwek,et al.  Multimodal Referential Acts in a Dialogue Game : from empirical investigations to algorithms , 2002 .

[24]  Yihsiu Chen,et al.  Language and Gesture: Lexical gestures and lexical access: a process model , 2000 .

[25]  Robbert-Jan Beun,et al.  Multimodal Reference to Objects: An Empirical Approach , 1998, Cooperative Multimodal Communication.

[26]  James A. Moore,et al.  Dialogue-Games: Metacommunication Structures for Natural Language Interaction , 1977, Cogn. Sci..

[27]  Emiel Krahmer,et al.  A new model for the generation of multimodal referring expressions , 2003 .

[28]  Ipke Wachsmuth,et al.  Deixis in Multimodal Human Computer Interaction: An Interdisciplinary Approach , 2003, Gesture Workshop.

[29]  Emiel Krahmer,et al.  The influence of target size and distance on the production of speech and gesture in multimodal referring expressions , 2004, INTERSPEECH.

[30]  Henrik Tramberend,et al.  Avocado: a distributed virtual reality framework , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[31]  David R. Traum,et al.  Conversational Actions and Discourse Situations , 1997, Comput. Intell..

[32]  G. Butterworth.  Pointing Is the Royal Road to Language for Babies , 2003 .

[33]  Marc Erich Latoschik,et al.  Resolving object references in multimodal dialogues for immersive virtual environments , 2004, IEEE Virtual Reality 2004.

[34]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[35]  Hannes Rieser,et al.  Pointing in Dialogue , 2004 .

[36]  Stefan Kopp,et al.  MURML: A Multimodal Utterance Representation Markup Language for Conversational Agents , 2002 .

[37]  H. H. Clark.  Pointing and placing. , 2003 .

[38]  D. McNeill.  Hand and Mind: What Gestures Reveal about Thought , 1992 .