论文信息 - Understanding Complex Visually Referring Utterances

Understanding Complex Visually Referring Utterances

We propose a computational model of visually-grounded spatial language understanding, based on a study of how people verbally describe objects in visual scenes. We describe our implementation of word level visually-grounded semantics and their embedding in a compositional parsing framework. The implemented system selects the correct referents in response to a broad range of referring expressions for a large percentage of test cases. In an analysis of the system's successes and failures we reveal how visual context influences the semantics of utterances and propose future extensions to the model that take such context into account.

Deb Roy | Peter Gorniak | Peter Gorniak | D. Roy

[1] Deb K. Roy,et al. Learning visually grounded words and syntax for a scene description task , 2002, Comput. Speech Lang..

[2] B. Landau,et al. “What” and “where” in spatial language and spatial cognition , 1993 .

[3] Terry Winograd,et al. Procedures As A Representation For Data In A Computer Program For Understanding Natural Language , 1971 .

[4] Barbara H. Partee,et al. Lexical semantics and compositionality. , 1995 .

[5] Alex Pentland,et al. Learning words from sights and sounds: a computational model , 2002, Cogn. Sci..

[6] Deb Roy,et al. A trainable spoken language understanding system for visual object selection , 2002, INTERSPEECH.

[7] Laura A. Carlson,et al. Grounding spatial language in perception: an empirical and computational investigation. , 2001, Journal of experimental psychology. General.

[8] Jeffrey Mark Siskind,et al. Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic , 1999, J. Artif. Intell. Res..

[9] Roger K. Moore. Computer Speech and Language , 1986 .

[10] James Pustejovsky,et al. The Generative Lexicon , 1995, CL.

[11] James F. Allen. Natural language understanding (2nd ed.) , 1995 .

[12] Jitendra Malik,et al. Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.