The identification of index terms in natural language object descriptions

"The flowering part, it looks like someone is sticking their tongue out" (a subject's description of Arethusa bulbosa, see Figure 1). The mechanisms that people use in natural settings to describe objects to one another can be used to inform the design of image retrieval and museum systems. The image retrieval problem may be recast as an object description problem where the images are of objects. This study examines the vocabulary and communication constructs that are used by novices and domain experts to describe objects in an object identification task. These human-centered devices may prove to be more understandable and easier to use than some purely computational approaches. The experimental conditions mimic a scenario where a person queries an agent (active botanical information resource) in natural language in order to identify plant images. The analysis identified the objects of discourse (objects, parts and relations) including analogies, exemplars, prototypical shapes and shape modification predicates such as "longer," and "wider." In spoken language novices and horticulturists use descriptive mechanisms similar to that in botanical text but at different frequencies. For example, participants rely heavily on visual analogies to objects both within and outside of the domain. "This looks like a X" where X is a plant (i.e. "daisy") or a non-plant (i.e. "butterfly" or "child's drawing of the sun"). The results suggest that indexing and retrieval systems should provide semantic level similarity mechanisms to allow for whole-object as well as part-wise visual analogy. The systems should also provide a visual vocabulary, a set of images that represent prototypes of the verbal terms collected in this study. Listener's Set[1] Speaker's Set[2] Figure 1: Arethusa bulbosa (Dragon's-mouth)