Semantic Image Retrieval based on Ontology and Relevance Model - A Preliminary Study

Modern Web search engines index hundreds of millions of images. To search these images is a daunting task for the user who can, realistically, only visually inspect a handful. In general, the way the user responds to an information need depends on the task at hand. Further, some tasks will require browsing, while others are targeted and require a more directed approach. In this paper, we present the preliminary results of research to develop a framework for applying semantics to enhance image retrieval. We consider this problem on two separate levels. First, we consider the application of an Ontology to define the semantic query space for image search and navigation, as well as to approximate the users context for the search. Secondly, in order to further improve upon the search results we apply the Relevance model, using data from the web to train the model. The role of the Relevance Model is to rank images from the search engine. The study also investigates how application of an ontology affects the quantity and quality the retrieved images and also the effects to the exp erience of the user in image search. We then contrast the results with those obtained by the Relevance Model for exactly similar search terms. The Relevance Model is based on a probabilistic model, which applies user definable language models to the text linking to the image. In our Relevance Model, the relevance of a HTML document linking to an image is evaluated and assigned with respect to highly ranked textual documents from the web. The ranking of the HTML document, is also assigned to ranking of the respective image. The main advantage here is that the Relevance Model can be learnt from the Web without any preparation of training data and is independent of the underlying algorithm of the image search engines. We show that navigation is indeed a very powerful tool for image browsing and that using the ontology dramatically enhances recall for specialized terms. Relevance feedback mainly improves precision by effective re-ranking. Keyword: Web Image Search, Language Model, Relevance Model, Wordnet, Ontology