References to graphical objects in interactive multimodal queries

Traditionally, interactive natural language systems assume a semantic model in which the entities referred to are in some abstract representation of a real or imagined world. In a system where graphical objects such as diagrams may be on the screen during the language interaction, there is possibility that the user may want to allude to these visual entities. Since graphical objects have their own properties (colour, shape, position on the screen, etc.) but may also represent items in a knowledge base which have other associated properties (price, geographical location, technical specifications, etc.), some systematic way is needed to enable such objects to be referred to in terms of either their screen properties or their associated attributes from the domain under discussion. In this paper, we present a formalisation for these arrangements, and show how our logical definitions can be used to generate constraints suitable for reference resolution within a natural language interpreter.

[1]  Michael D. Wilson,et al.  Multimodal integration in MMI2: anaphora resolution and mode selection , 1992 .

[2]  Jungyun Seo,et al.  Resolution of referring expressions in a Korean multimodal dialogue system , 2003, TALIP.

[3]  Marilyn A. Walker,et al.  Centering, Anaphora Resolution, and Discourse Structure , 1997, ArXiv.

[4]  Luis Alberto Pineda,et al.  A Model for Multimodal Reference Resolution , 2000, Computational Linguistics.

[5]  Michael Halliday,et al.  An Introduction to Functional Grammar , 1985 .

[6]  Stuart C. Shapiro,et al.  Intelligent Multi-Media Interface Technology , 1988, SGCH.

[7]  Graeme D. Ritchie.  Computer modelling of English grammar , 1977 .

[8]  Helmut Horacek,et al.  An Algorithm for Generating Referential Descriptions with Flexible Interfaces , 1997, ACL.

[9]  Carl Pollard,et al.  A Centering Approach to Pronouns , 1987, ACL.

[10]  Allan Ramsay.  Reference and Computation: An Essay in Applied Philosophy of Language , 1991 .

[11]  William A. Woods,et al.  Semantics For a Question-Answering System , 1967, Outstanding Dissertations in the Computer Sciences.

[12]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[13]  G. Frege.  On Sense and Reference , 1948 .

[14]  Candace L. Sidner,et al.  Focusing in the comprehension of definite anaphora , 1986 .

[15]  Terry Winograd,et al.  Understanding natural language , 1974 .

[16]  Mark T. Maybury,et al.  Intelligent multimedia interfaces , 1994, CHI Conference Companion.

[17]  Kees van Deemter,et al.  Towards a Logic of Ambiguous Expressions 1 Coping with Ambiguous Information , 1996 .

[18]  William A. Woods,et al.  Semantics and Quantification in Natural Language Question Answering , 1986, Adv. Comput..

[19]  Peter Thanisch,et al.  Natural language interfaces to databases – an introduction , 1995, Natural Language Engineering.

[20]  C. S. Mellish.  Computer interpretation of natural language descriptions , 2000 .

[21]  Robert Dale,et al.  Generating Referring Expressions Involving Relations , 1991, EACL.

[22]  Candace L. Sidner,et al.  Focusing for Interpretation of Pronouns , 1981, CL.

[23]  Norbert Reithinger,et al.  XTRA: A Natural-Language Access System to Expert Systems , 1989, Int. J. Man Mach. Stud..

[24]  Scott Weinstein,et al.  Centering: A Framework for Modeling the Local Coherence of Discourse , 1995, CL.

[25]  Philip R. Cohen,et al.  Synergistic use of direct manipulation and natural language , 1989, CHI '89.

[26]  Kees van Deemter,et al.  Semantic ambiguity and underspecification , 1996 .

[27]  Sharon L. Oviatt,et al.  Unification-based Multimodal Integration , 1997, ACL.

[28]  Shalom Lappin,et al.  An Algorithm for Pronominal Anaphora Resolution , 1994, CL.

[29]  Joyce Yue Chai,et al.  Semantics-based Representation for Multimodal Interpretation in Conversational Systems , 2002, COLING.

[30]  Margaret King,et al.  Parsing Natural Language , 1983 .

[31]  Jeannette G. Neal,et al.  Multi-Modal References in Human-Computer Dialogue , 1988, AAAI.

[32]  H. E. Shrobe,et al.  Exploring Artificial Intelligence , 1988 .

[33]  M. Walker,et al.  Centering Theory in Discourse , 1998 .

[34]  W. Levelt,et al.  Speaking: From Intention to Articulation , 1990 .

[35]  Gilles Fauconnier,et al.  Mental Spaces: Aspects of Meaning Construction in Natural Language , 1985 .

[36]  Louisa Sadler,et al.  Structural Non-Correspondence in Translation , 1991, EACL.

[37]  Steven K. Feiner,et al.  Generating Cross-References for Multimedia Explanation , 1992, AAAI.

[38]  Alan K. Mackworth.  Consistency in Networks of Relations , 1977, Artif. Intell..

[39]  Thomas Rist,et al.  Referring To World Objects With Text And Pictures , 1994, COLING.

[40]  Karen Spärck Jones,et al.  Natural language interfaces to databases , 1990, The Knowledge Engineering Review.

[41]  Nicholas John Haddock.  Incremental semantics and interactive syntactic processing , 1988 .

[42]  Michael Johnston,et al.  Unification-based Multimodal Parsing , 1998, ACL.

[43]  Patrick Henry Winston,et al.  The psychology of computer vision , 1976, Pattern Recognit..

[44]  Renata Vieira,et al.  Definite Description Processing in Unrestricted Text , 1998 .

[45]  William A. Woods,et al.  Procedural semantics for a question-answering machine , 1899, AFIPS Fall Joint Computing Conference.

[46]  Thomas Rist,et al.  The Design of Illustrated Documents as a Planning Task , 1993, AAAI Workshop on Intelligent Multimedia Interfaces.

[47]  G. Nunberg.  The non-uniqueness of semantic solutions: Polysemy , 1979 .

[48]  Jerry R. Hobbs.  Resolving pronoun references , 1986 .

[49]  David G. Novick,et al.  Natural-language interfaces , 2000, CHI Extended Abstracts.