Quantitative assessment of image retrieval effectiveness

Content-based retrieval (CBR) promises to greatly improve capabilities for searching for images based on semantic features and visual appearance. However, developing a framework for evaluating image retrieval effectiveness remains a significant challenge. Difficulties include determining how matching at different description levels affects relevance, designing meaningful benchmark queries of large image collections, and developing suitable quantitative metrics for measuring retrieval effectiveness. This article studies the problems of developing a framework and testbed for quantitative assessment of image retrieval effectiveness. In order to better harness the extensive research on CBR and improve capabilities of image retrieval systems, this article advocates the establishment of common image retrieval testbeds consisting of standardized image collections, benchmark queries, relevance assessments, and quantitative evaluation methods.

[1]  Shih-Fu Chang,et al.  Transform features for texture classification and discrimination in large image databases , 1994, Proceedings of 1st International Conference on Image Processing.

[2]  N. R. Howe,et al.  Using artificial queries to evaluate image retrieval , 2000, 2000 Proceedings Workshop on Content-based Access of Image and Video Libraries.

[3]  Joshua R. Smith,et al.  Image retrieval evaluation , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[4]  James Lee Hafner,et al.  Efficient Color Histogram Indexing for Quadratic Form Distance Functions , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Shih-Fu Chang,et al.  Visually Searching the Web for Content , 1997, IEEE Multim..

[6]  Serge J. Belongie,et al.  Region-based image querying , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[7]  Stephen W. Smoliar,et al.  Content based video indexing and retrieval , 1994, IEEE MultiMedia.

[8]  S. Sclaroff,et al.  ImageRover: a content-based image browser for the World Wide Web , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[9]  R.M. Haralick,et al.  Statistical and structural approaches to texture , 1979, Proceedings of the IEEE.

[10]  Laura M. Haas,et al.  Querying Multimedia Data from Multiple Repositories by Content: the Garlic Project , 1995, VDB.

[11]  Phil Brodatz,et al.  Textures: A Photographic Album for Artists and Designers , 1966 .

[12]  Alex Pentland,et al.  Photobook: tools for content-based manipulation of image databases , 1994, Electronic Imaging.

[13]  Michael Stonebraker,et al.  Chabot: Retrieval from a Relational Database of Images , 1995, Computer.

[14]  Terry E. Weymouth,et al.  Semantic Queries with Pictures: The VIMSYS Model , 1991, VLDB.

[15]  Shih-Fu Chang,et al.  Integrated spatial and feature image query , 1999, Multimedia Systems.

[16]  Alfred V. Aho,et al.  Columbia digital news project: an environment for briefing and search over multimedia information , 1998, International Journal on Digital Libraries.

[17]  N. Mwara,et al.  Picture retrieval by content description , 1992, J. Inf. Sci..

[18]  Donna K. Harman,et al.  Overview of the First Text REtrieval Conference (TREC-1) , 1992, TREC.

[19]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[20]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[21]  John R. Smith,et al.  Conceptual modeling of audio-visual content , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[22]  Vijay V. Raghavan,et al.  Design and evaluation of algorithms for image retrieval by spatial similarity , 1995, TOIS.