Information Fusion for Visual Reference Resolution in Dynamic Situated Dialogue

Human-Robot Interaction (HRI) invariably involves dialogue about objects in the environment in which the agents are situated. The paper focuses on the issue of resolving discourse references to such visual objects. The paper addresses the problem using strategies for intra-modal fusion (identifying that different occurrences concern the same object), and inter-modal fusion, (relating object references across different modalities). Core to these strategies are sensorimotoric coordination, and ontology-based mediation between content in different modalities. The approach has been fully implemented, and is illustrated with several working examples.

[1]  D. Byron.  Understanding Referring Expressions in Situated Language Some Challenges for Real-World Agents Donna , 2003 .

[2]  Heiner Stuckenschmidt,et al.  Ontology-Based Integration of Information - A Survey of Existing Approaches , 2001, OIS@IJCAI.

[3]  Jason Baldridge,et al.  Multi-Modal Combinatory Categorial Grammar , 2003, EACL.

[4]  John D. Kelleher,et al.  A Context-Dependent Model of Proximity in Physically Situated Environments , 2005 .

[5]  James F. Allen,et al.  An architecture for a generic dialogue shell , 2000, Natural Language Engineering.

[6]  Adam Cheyer,et al.  The Open Agent Architecture , 1997, Autonomous Agents and Multi-Agent Systems.

[7]  Iryna Gurevych,et al.  Less is More: Using a single knowledge representation in dialogue systems , 2003, HLT-NAACL 2003.

[8]  Alessandro Saffiotti,et al.  Maintaining Coherent Perceptual Information Using Anchoring , 2005, IJCAI.

[9]  Massimo Poesio,et al.  Discourse interpretation and the scope of operators , 1994 .

[10]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[11]  Alessandro Saffiotti,et al.  An introduction to the anchoring problem , 2003, Robotics Auton. Syst..

[12]  John D. Kelleher,et al.  Structural descriptions in human-assisted robot visual learning , 2006, HRI '06.

[13]  Alex Lascarides,et al.  Logics of Conversation , 2005, Studies in natural language processing.

[14]  Jason Baldridge,et al.  Coupling CCG and Hybrid Logic Dependency Semantics , 2002, ACL.

[15]  Johan Bos,et al.  Meaningful Conversation with a Mobile Robot , 2003, EACL.