Evaluating strategies and systems for content based indexing of person images on the Web

Content based indexing of multimedia has always been a challenging task. The enormity and the diversity of the multimedia content on the web adds another dimension to this challenge. In this paper, we examine ways of combining visual and textual information for content based indexing of multimedia on the web. In particular, we examine different methods of combining evidences due to face detection, Text/HTML analysis and face recognition for identifying person images. We provide experimental evaluation of the following strategies: i) Face detection on the image followed by Text/HTML analysis of the containing page; ii) face detection followed by face recognition; iii) face detection followed by a linear combination of evidences due to text/HTML analysis and face recognition; and iv) face detection followed by a Dempster-Shafer combination of evidences due to text/HTML analysis and face recognition. These strategies were implemented in an automatic web search agent named Diogenes1 and compared against some well known web image search engines. The latter includes commercial systems such as Alta Vista, Lycos and Ditto, and a research prototype, WebSEEk. We report the results of our experimental retrievals where Diogenes outperformed these search engines for celebrity image queries in terms of average precision.