Towards Situated Dialogue: Revisiting Referring Expression Generation

In situated dialogue, humans and agents have mismatched capabilities of perceiving the shared environment. Their representations of the shared world are misaligned. Thus referring expression generation (REG) will need to take this discrepancy into consideration. To address this issue, we developed a hypergraph-based approach to account for group-based spatial relations and uncertainties in perceiving the environment. Our empirical results have shown that this approach outperforms a previous graph-based approach with an absolute gain of 9%. However, while these graph-based approaches perform effectively when the agent has perfect knowledge or perception of the environment (e.g., 84%), they perform rather poorly when the agent has imperfect perception of the environment (e.g., 45%). This big performance gap calls for new solutions to REG that can mediate a shared perceptual basis in situated dialogue.

[1]  Changsong Liu,et al.  Modeling Collaborative Referring for Situated Referential Grounding , 2013, SIGDIAL Conference.

[2]  C. K. Ogden.  A Source Book Of Gestalt Psychology , 2013 .

[3]  Kees van Deemter,et al.  Generating Expressions that Refer to Visible Objects , 2013, NAACL.

[4]  Changsong Liu,et al.  Towards Mediating Shared Perceptual Basis in Situated Dialogue , 2012, SIGDIAL Conference.

[5]  Stefan Kopp,et al.  Referring in Installments: A Corpus Study of Spoken Object References in an Interactive Virtual Environment , 2012, INLG.

[6]  Emiel Krahmer,et al.  Computational Generation of Referring Expressions: A Survey , 2012, CL.

[7]  Kees van Deemter,et al.  Two Approaches for Generating Size Modifiers , 2011, ENLG.

[8]  Emiel Krahmer,et al.  Does Size Matter - How Much Data is Required to Train a REG Algorithm? , 2011, ACL.

[9]  Alfons Maes,et al.  Overspecification facilitates object identification , 2011 .

[10]  Stefan Weijers.  Referring Expressions with Groups as Landmarks , 2011 .

[11]  Dan Klein,et al.  A Game-Theoretic Approach to Generating Spatial Descriptions , 2010, EMNLP.

[12]  Albert Gatt,et al.  The TUNA-REG Challenge 2009: Overview and Evaluation Results , 2009, ENLG.

[13]  David Schlangen,et al.  A Simple Method for Resolution of Definite Reference in a Shared Visual Context , 2008, SIGDIAL Workshop.

[14]  Albert Gatt,et al.  Attribute Selection for Referring Expression Generation: New Algorithms and Evaluation Methods , 2008, INLG.

[15]  Robert Dale,et al.  The Use of Spatial Relations in Referring Expression Generation , 2008, INLG.

[16]  Iris Hendrickx,et al.  GRAPH: The Costs of Redundancy in Referring Expressions , 2008, INLG.

[17]  Emiel Krahmer,et al.  Cost-based attribute selection for GRE (GRAPH-SC/GRAPH-FP) , 2007 .

[18]  Ielka van der Sluis,et al.  Evaluating algorithms for the Generation of Referring Expressions using a balanced corpus , 2007, ENLG.

[19]  Madalina Croitoru,et al.  A Conceptual Graph Approach for the Generation of Referring Expressions , 2007, IJCAI.

[20]  S. Harnad.  Symbol grounding problem , 1990, Scholarpedia.

[21]  John D. Kelleher,et al.  Incremental Generation of Spatial Referring Expressions in Situated Dialog , 2006, ACL.

[22]  Takenobu Tokunaga,et al.  Group-Based Generation of Referring Expressions , 2006, INLG.

[23]  G. Sauder.  Does size matter , 2006 .

[24]  Albert Gatt,et al.  Structuring Knowledge for Reference Generation: a Clustering Algorithm , 2022 .

[25]  Helmut Horacek.  Generating Referential Descriptions Under Conditions of Uncertainty , 2005, ENLG.

[26]  Deb Roy,et al.  Grounded Semantic Composition for Visual Scenes , 2011, J. Artif. Intell. Res..

[27]  Takenobu Tokunaga,et al.  Generation of Relative Referring Expressions based on Perceptual Grouping , 2004, COLING.

[28]  Emiel Krahmer,et al.  Graph-Based Generation of Referring Expressions , 2003, CL.

[29]  Philip R. Cohen,et al.  Referring as a Collaborative Process , 2003 .

[30]  Sheel Sanjay Dhande,et al.  A computational model to connect gestalt perception and natural language , 2003 .

[31]  A. Meyer,et al.  Tracking the time course of multidimensional stimulus discrimination: Analyses of viewing patterns and processing times during “same”-“different“ decisions , 2002 .

[32]  Dengsheng Zhang,et al.  An integrated approach to shape based image retrieval , 2002 .

[33]  Marilyn A. Walker,et al.  Learning Attribute Selections for Non-Pronominal Expressions , 2000, ACL.

[34]  D. Roy.  Learning Visually Grounded Words and Syntax of Natural Spoken Language , 2000 .

[35]  Robert Dale,et al.  Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions , 1995, Cogn. Sci..

[36]  Philip Edmonds.  Collaboration On Reference To Objects That Are Not Mutually Known , 1994, COLING.

[37]  Giorgio Gallo,et al.  Directed Hypergraphs and Applications , 1993, Discret. Appl. Math..

[38]  R. Dale.  Generating referring expressions - constructing descriptions in a domain of objects and processes , 1995, ACL-MIT press series in natural language processing.

[39]  Graeme Hirst,et al.  Collaborating on Referring Expressions , 1991, CL.

[40]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[41]  Robert Dale,et al.  Cooking Up Referring Expressions , 1989, ACL.

[42]  W. D. Ellis.  A Source Book Of Gestalt Psychology , 1939 .

[43]  M. Wertheimer.  Laws of organization in perceptual forms. , 1938 .

[44]  A. Mojsilovi.  A Computational Model for Color Naming and Describing Color Composition of Images , 2022 .