Annotation Tools and Knowledge Representation for a Text-To-Scene System

Text-to-scene conversion requires knowledge about how actions and locations are expressed in language and realized in the world. To provide this knowlege, we are creating a lexical resource (VigNet) that extends FrameNet by creating a set of intermediate frames (vignettes) that bridge between the high-level semantics of FrameNet frames and a new set of low-level primitive graphical frames. Vignettes can be thought of as a link between function and form ‐ between what a scene means and what it looks like. In this paper, we describe the set of primitive graphical frames and the functional properties of 3D objects (affordances) we use in this decomposition. We examine the methods and tools we have developed to populate VigNet with a large number of action and location vignettes.

[1]  Martha Palmer,et al.  Class-Based Construction of a Verb Lexicon , 2000, AAAI/IAAI.

[2]  D. Norman.  The psychology of everyday things", Basic Books Inc , 1988 .

[3]  Richard Sproat,et al.  WordsEye: an automatic text-to-scene conversion system , 2001, SIGGRAPH.

[4]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[5]  Julia Hirschberg,et al.  Evaluating a text-to-scene generation system as an aid to literacy , 2011, SLaTE.

[6]  Ray Jackendoff.  Semantics and Cognition , 1983 .

[7]  Julia Hirschberg,et al.  Spatial Relations in Text-to-Scene Conversion , 2010 .

[8]  S. Greenberg,et al.  The Psychology of Everyday Things , 2012 .

[9]  三嶋 博之.  The theory of affordances , 2008 .

[10]  Richard Sproat.  Inferring the environment in a text-to-scene conversion system , 2001, K-CAP '01.

[11]  G. Miller,et al.  Language and Perception , 1976 .

[12]  Bob Coyne,et al.  Collecting Semantic Data from Mechanical Turk for a Lexical Knowledge Resource in a Text to Picture Generating System , 2011, IWCS.

[13]  Bob Coyne,et al.  Data collection and normalization for building the Scenario-Based Lexical Knowledge Resource of a text-to-scene conversion system , 2010, 2010 Fifth International Workshop Semantic Media Adaptation and Personalization.

[14]  Roger C. Schank,et al.  SCRIPTS, PLANS, GOALS, AND UNDERSTANDING , 1988 .

[15]  Alexis Nasr,et al.  MICA: A Probabilistic Dependency Parser Based on Tree Insertion Grammars (Application Note) , 2009, HLT-NAACL.

[16]  Richard Sproat,et al.  Collecting Spatial Information for Locations in a Text-to-Scene Conversion System , 2011 .

[17]  R. Shaw,et al.  Perceiving, Acting and Knowing : Toward an Ecological Psychology , 1978 .

[18]  Josef Ruppenhofer,et al.  FrameNet II: Extended theory and practice , 2006 .

[19]  Bob Coyne,et al.  VigNet: Grounding Language in Graphics using Frame Semantics , 2011, RELMS@ACL.

[20]  Bob Coyne,et al.  Collecting Semantic Information for Locations in the Scenario-Based Lexical Knowledge Resource of a Text-to-Scene Conversion System , 2011, KES.