Pictures from Words, Pictures from Text: Constructing Pictorial Representations of Meaning from Text

Abstract : This report describes a design study carried out at the Computing Research Laboratory, New Mexico State University. The goal was to investigate the feasibility of creating a visual interpretation of the spoken messages passed among the crew of an Unmanned Aerial Vehicle (UAV). Because of the limited domain of UAV crew dialogs, it seemed possible to produce visual representations of their messages. These visualized messages could be used for communication among UAV crew members who do not speak the same language, and they could also visually verify any spoken communications. The following tasks were carried out by the research team: (1) a general analysis of how images could be used to represent dialogs; (2) a more specific analysis of dialogs and parse trees produced from experimental UAV dialogs; (3) participation in discussions and in a simulated flight as crew members while visiting researchers at AFRL, Mesa; (4) development of a set of icons and conventions to represent mission status and the form of crew member dialogs; (5) design of a prototype visualization system -- Chat2Pix; (6) implementation of the prototype using Visual Basic to run on Windows systems; (7) redesign of the prototype to display more structured information; and (8) implementation and delivery of a second version of the visualization system -- Chat2Pix2.

[1]  Sergei Nirenburg,et al.  A Situated Ontology for Practical NLP , 1995 .

[2]  G. Lakoff,et al.  Women, Fire, and Dangerous Things: What Categories Reveal about the Mind , 1988 .

[3]  Sergei Nirenburg,et al.  MOQA - Meaning-Oriented Question-Answering , 2004, RIAO.

[4]  姚小平,et al.  语言学简史 : [英文版] = A Short History of Linguistics , 1969 .

[5]  Jerry T. Ball,et al.  A Cognitively Plausible Model of Language Comprehension , 2004 .

[6]  David Farwell,et al.  Translation Differences and Pragmatics-Based MT , 1998, AMTA.

[7]  Sergei Nirenburg,et al.  Semantic Classification for Practical Natural Language Processing , 1995 .

[8]  Peter W. Foltz,et al.  Automated Team Discourse Modeling: Test of Performance and Generalization , 2006 .

[9]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[10]  Eleanor Rosch,et al.  Principles of Categorization , 1978 .

[11]  Sergei Nirenburg,et al.  Book Review: Ontological Semantics, by Sergei Nirenburg and Victor Raskin , 2004, CL.

[12]  Rada Mihalcea,et al.  PicNet: Augmenting Semantic Resources with Pictorial Representations , 2005, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[13]  Robert D. Rodman,et al.  An Introduction to Language , 1984 .

[14]  Roger C. Schank,et al.  SCRIPTS, PLANS, GOALS, AND UNDERSTANDING , 1988 .

[15]  Raffaele Simone.  Iconicity in Language , 1995 .

[16]  Rada Mihalcea,et al.  Toward communicating simple sentences using pictorial representations , 2008, AMTA.

[17]  Nizar Habash,et al.  Interlingual Annotation of Multilingual Text Corpora , 2004, FCP@NAACL-HLT.

[18]  F. Saussure,et al.  Course in General Linguistics , 1960 .

[19]  E. Tufte.  Beautiful Evidence , 2006 .

[20]  Ferdinand de Saussure.  Course in General Linguistics , 1916 .