Learning when to point: A data-driven approach

The relationship between how people describe objects and when they choose to point is complex and likely to be influenced by factors related to both perceptual and discourse context. In this paper, we explore these interactions using machine-learning on a dialogue corpus, to identify multimodal referential strategies that can be used in automatic multimodal generation. We show that the decision to use a pointing gesture depends on features of the accompanying description (especially whether it contains spatial information), and on visual properties, especially distance or separation of a referent from its previous referent.

[1]  Albert Gatt,et al.  What and Where: An Empirical Investigation of Pointing Gestures and Descriptions in Multimodal Referring Actions , 2013, ENLG.

[2]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[3]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[4]  Emiel Krahmer,et al.  Graph-Based Generation of Referring Expressions , 2003, CL.

[5]  N. J. Enfield,et al.  The Anatomy of Meaning: Speech, Gesture, and Composite Utterances , 2009 .

[6]  Sotaro Kita,et al.  What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking , 2003 .

[7]  D. McNeill.  So you think gestures are nonverbal , 1985 .

[8]  Ipke Wachsmuth,et al.  Incremental Generation of Multimodal Deixis Referring to Objects , 2005, ENLG.

[9]  Robert Dale,et al.  Cooking Up Referring Expressions , 1989, ACL.

[10]  Jan Peter de Ruiter,et al.  The Interplay Between Gesture and Speech in the Production of Referring Expressions: Investigating the Tradeoff Hypothesis , 2012, Top. Cogn. Sci..

[11]  Robbert-Jan Beun,et al.  Object reference in a shared domain of conversation , 1998 .

[12]  Ielka van der Sluis,et al.  Towards a Balanced Corpus of Multimodal Referring Expressions in Dialogue , 2008 .

[13]  Anne H. Anderson,et al.  The Hcrc Map Task Corpus , 1991 .

[14]  A. Bangerter,et al.  Using Pointing and Describing to Achieve Joint Focus of Attention in Dialogue , 2004, Psychological science.

[15]  Susan Duncan,et al.  Growth points in thinking-for-speaking , 1998 .

[16]  Thomas Rist,et al.  Coping with Temporal Constraints in Multimedia Presentation Planning , 1996, AAAI/IAAI, Vol. 1.

[17]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[18]  Paul Piwek,et al.  Modality Choice for Generation of Referring Acts: Pointing versus Describing , 2007 .

[19]  Emiel Krahmer,et al.  Computational Generation of Referring Expressions: A Survey , 2012, CL.