Intelligent Information Retrieval for a Multimedia Database Using Captions

This report describes an intelligent information retrieval system, MARIE, that employs natural language processing techniques for indexing and retrieving multimedia data. Captions describe photographs from the Naval Air Warfare Center Weapons Division, China Lake, California; the captions were written in English consisting mostly of noun phrases. For our work, an object-oriented type hierarchy represents semantic knowledge. Captions are parsed to produce a logical form, from which nouns and verbs are extracted to form keyword files. User queries are also specified in natural language. A two-phase match process is employed between the query and database. A coarse-grain match searches the keyword files and issues SQL queries to a relational database as necessary to find candidate captions for further analysis. A fine-grain match then compares the logical form of the query to the logical form for each caption. A list of caption IDs and accompanying match scores is then presented to the user, who can view the image and supporting data. A companion technical report contains the programs for the system discussed here.