A New Model for Generating Multimodal Referring Expressions

We present a new algorithm for the generation of multimodal referring expressions (combining language and deictic gestures).1 The approach differs from earlier work in that we allow for various gradations of preciseness in pointing, ranging from unambiguous to vague pointing gestures. The model predicts that linguistic properties realized in the generated expression are co-dependent on the kind of pointing gesture included. The decision to point is based on a tradeoff between the costs of pointing and the costs of linguistic properties, where both kinds of costs are computed in empirically motivated ways. The model has been implemented using a graph-based generation algorithm.

[1]  Kees van Deemter.  Generating Referring Expressions: Boolean Extensions of the Incremental Algorithm , 2002, CL.

[2]  Carla Huls,et al.  Automatic Referent Resolution of Deictic and Anaphoric Expressions , 1995, CL.

[3]  Ian Mackenzie,et al.  Fitts' law as a performance model in human-computer interaction , 1992 .

[4]  Philip R. Cohen.  The Pragmatics of Referring and the Modality of Communication , 1984, Comput. Linguistics.

[5]  Wim Claassen,et al.  Generating Referring Expressions in a Multimodal Environment , 1992, NLG.

[6]  Donia Scott,et al.  Book Reviews: Generating Referring Expressions , 1994, CL.

[7]  Claire Gardent,et al.  Generating Minimal Definite Descriptions , 2002, ACL.

[8]  Kees van Deemter.  Generating Referring Expressions: Beyond the Incremental Algorithm , 2000 .

[9]  Thomas Rist,et al.  Coping with Temporal Constraints in Multimedia Presentation Planning , 1996, AAAI/IAAI, Vol. 1.

[10]  I. I. van der Sluis,et al.  Generating Referring Expressions in a Multimodal Context , 2001 .

[11]  E. Bizzi,et al.  Muscle properties and the control of arm movements , 1990 .

[12]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[13]  Emiel Krahmer,et al.  Generating Referring Expressions in a Multimodal Context An empirically oriented approach , 2000, CLIN.

[14]  James C. Lester,et al.  Deictic Believability: Coordinated Gesture, Locomotion, and Speech in Lifelike Pedagogical Agents , 1999, Appl. Artif. Intell..

[15]  Emiel Krahmer,et al.  Graph-Based Generation of Referring Expressions , 2003, CL.

[16]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[17]  Robbert-Jan Beun,et al.  Object reference in a shared domain of conversation , 1998 .

[18]  A. Wing,et al.  The Psychology of human movement , 1984 .