A web-based evaluation system for CBIR

This papers describes a benchmark test for content-based image retrieval systems (CBIRSs) with the query by example (QBE) query paradigm. This benchmark is accessible via the Internet and thus allows to evaluate any CBIRS which is compliant with Multimedia Retrieval Markup Language (MRML) for query formulation and result transmission. Thus it allows a quick and easy comparison between different features and algorithms for CBIRSs. The benchmark is based on a standardized communication protocol to do the communication between benchmark server and benchmarked system and it uses a freely downloadable image database (DB) to make the results reproducible. A CBIR system that uses MRML and other components to develop MRML-based applications can be downloaded free of charge as well. The evaluation is based on several queries and known relevance sets for these queries. Several answer sets for one query image are possible if judgments of several users exist, thus almost any sort of judgment can be incorporated into the system. The final results are averaged over all queries. The evaluation of several steps of relevance feedback (RF) based on the collected relevance judgments is also included into the benchmark. The performance of RF is often regarded to be even more important than the performance in the first query step because only with RF the adaptation of the system to the users subjective goal can be measured. For the evaluation of a system with RF, the same evaluation measures are used as for the first query step.

[1]  Ellen M. Voorhees,et al.  Overview of the seventh text retrieval conference (trec-7) [on-line] , 1999 .

[2]  Ellen M. Voorhees,et al.  Overview of the Seventh Text REtrieval Conference , 1998 .

[3]  Thierry Pun,et al.  Performance evaluation in content-based image retrieval: overview and proposals , 2001, Pattern Recognit. Lett..

[4]  Clement H. C. Leung,et al.  Benchmarking for Content-Based Visual Information Search , 2000, VISUAL.

[5]  Alexander Dimai.  Assessment of Effectiveness of Content Based Image Retrieval Systems , 1999, VISUAL.

[6]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[7]  Thierry Pun,et al.  Toward a fair benchmark for image browsers , 2000, SPIE Optics East.

[8]  Erkki Oja,et al.  Evaluating the Performance of Content-Based Image Retrieval Systems , 2000, VISUAL.

[9]  Henning Müller,et al.  Relevance Feedback and Term Weighting Schemes for Content-Based Image Retrieval , 1999, VISUAL.

[10]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[11]  Thierry Pun,et al.  A Comparison of Human and Machine Assessments of Image Similarity for the Organization of Image Databases , 1997 .

[12]  Ingemar J. Cox,et al.  Target testing and the PicHunter Bayesian multimedia retrieval system , 1996, Proceedings of the Third Forum on Research and Technology Advances in Digital Libraries,.

[13]  Cyril W. Cleverdon,et al.  Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 2, Appendices , 1966 .

[14]  Corinne Jörgensen.  Classifying Images: Criteria for Grouping as Revealed in a Sorting Task , 1995 .

[15]  Neil J. Gunther,et al.  Benchmark for image retrieval using distributed systems over the Iinternet: BIRDS-I , 2000, IS&T/SPIE Electronic Imaging.

[16]  Thierry Pun,et al.  MRML: A Communication Protocol for Content-Based Image Retrieval , 2000, VISUAL.