Evaluation and Benchmarking of Image Database Navigation Tools

Evaluation and benchmarking in content-based image retrieval has always been a somewhat neglected research area, making it difficult to judge the efficacy of many presented approaches. In this paper we investigate the issue of benchmarking for image database navigation tools. Such systems allow the browsing of large image collections in an intuitive manner enabling the user to find or annotate images effectively and efficiently. We present previous work aimed at testing such systems and propose a set of guidelines for a standardised benchmark that can be used to evaluate and compare different image browsing approaches. We argue that such a benchmark needs to be based on a standardised image database, and should include user tasks such as target and category search, and annotation as well as record the opinion of the users. The performance on each of the selected tasks needs to be quantifiable by a defined set of measures in order to allow for comparison between systems.