The role of evaluation in the development of content-based retrieval techniques

There are no satisfactory methods for measuring the effectiveness of multimedia search techniques. Precision and recall types of metrics have been used in some of the literature but are impractical due to the tedious process of measuring relevances. The process is even more complicated because of human subjectivity in tasks involving multimedia. Also, multimedia collections quickly grow very large, making evaluation expensive with respect to the required hardware. There seem to be no standard corpora or benchmark procedures. In this document, we review the evaluation approaches that have been taken in the literature about multimedia retrieval systems. Our review covers far from all related publications, but we believe the sample of papers studied is sufficiently large to draw our (sad) conclusions.

[1]  Peter Schäuble,et al.  Speech Retrieval Based on Automatic Indexing , 1995, MIRO.

[2]  T.S. Huang,et al.  A relevance feedback architecture for content-based multimedia information retrieval systems , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[3]  Thomas S. Huang,et al.  Relevance feedback techniques in interactive content-based image retrieval , 1997, Electronic Imaging.

[4]  De Bonet,et al.  Novel statistical multiresolution techniques for image synthesis, discrimination, and recognition , 1997 .

[5]  A. Desai Narasimhalu,et al.  Experiences with Content Based Retrieval of Multimedia Information , 1995, MIRO.

[6]  Simone Santini,et al.  Similarity Matching , 1995, ACCV.

[7]  Paul A. Viola,et al.  Texture recognition using a non-parametric multi-scale statistical model , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[8]  Tom Minka,et al.  Interactive learning with a "society of models" , 1997, Pattern Recognit..

[9]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[10]  Aidong Zhang,et al.  Efficient resource selection in distributed visual information systems , 1997, MULTIMEDIA '97.

[11]  Thomas P. Minka,et al.  An image database browser that learns from user interaction , 1996 .

[12]  Karen Spärck Jones,et al.  Open-vocabulary speech indexing for voice and video mail retrieval , 1997, MULTIMEDIA '96.

[13]  W.E.L. Grimson,et al.  Training templates for scene classification using a few examples , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[14]  Wolfgang Effelsberg,et al.  Video abstracting , 1997, CACM.

[15]  Shih-Fu Chang,et al.  Quad-tree segmentation for texture-based image query , 1994, MULTIMEDIA '94.

[16]  Christos Faloutsos,et al.  QBIC project: querying images by content, using color, texture, and shape , 1993, Electronic Imaging.

[17]  Kris Popat,et al.  Cluster-based probability model and its application to image and texture processing , 1997, IEEE Trans. Image Process..

[18]  Tat-Seng Chua,et al.  Content-based retrieval of segmented images , 1994, MULTIMEDIA '94.

[19]  Aidong Zhang,et al.  Block-oriented image decomposition and retrieval in image database systems , 1996, Proceedings of International Workshop on Multimedia Database Management Systems.

[20]  Michael Stonebraker,et al.  Chabot: Retrieval from a Relational Database of Images , 1995, Computer.

[21]  C.-C. Jay Kuo,et al.  Joint spatial-spectral indexing for image retrieval , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[22]  John W. Fisher,et al.  Flexible histograms: a multiresolution target discrimination model , 1998, Defense, Security, and Sensing.

[23]  John R. Smith,et al.  Intelligent multimedia information retrieval , 1997 .

[24]  Thomas S. Huang,et al.  Supporting similarity queries in MARS , 1997, MULTIMEDIA '97.

[25]  S. Sclaroff,et al.  ImageRover: a content-based image browser for the World Wide Web , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[26]  Martin L. Kersten,et al.  The Acoi Algebra: a Query Algebra for Image Retrieval Systems , 1998, BNCOD.

[27]  C.-C. Jay Kuo,et al.  Content-based image retrieval using multiresolution histogram representation , 1995, Other Conferences.

[28]  Fang Liu,et al.  Periodicity, Directionality, and Randomness: Wold Features for Image Modeling and Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Jonathan Foote,et al.  A Similarity Measure for Automatic Audio Classification , 1997 .

[30]  Thomas S. Huang,et al.  Automatic Matching Tool Selection Using Relevance Feedback In Mars , 1997 .

[31]  Alistair Sutcliffe,et al.  Empirical studies in multimedia information retrieval , 1997 .

[32]  Yong Rui,et al.  Multimedia Analysis and Retrieval System , 1997 .

[33]  Mark T. Maybury,et al.  Towards content-based browsing of broadcast news video , 1997 .

[34]  Shih-Fu Chang,et al.  VideoQ: an automated content based video search system using visual cues , 1997, MULTIMEDIA '97.

[35]  Marti A. Hearst.  Multi-Paragraph Segmentation Expository Text , 1994, ACL.

[36]  Takeo Kanade,et al.  elligent Access Video: formedia Project , 1996 .

[37]  Rosalind W. Picard,et al.  Interactive Learning Using a "Society of Models" , 2017, CVPR 1996.

[38]  Karen Spärck Jones,et al.  Retrieving spoken documents by combining multiple index sources , 1996, SIGIR '96.

[39]  Arnold W. M. Smeulders,et al.  A comparative study of several color models for color image invariant retrieval , 1996 .

[40]  Aidong Zhang,et al.  SemQuery: Semantic Clustering and Querying on Heterogeneous Features for Visual Data , 2002, IEEE Trans. Knowl. Data Eng..

[41]  Aidong Zhang,et al.  NeuroMerge: an approach for merging heterogeneous features in content-based image retrieval systems , 1998, Proceedings International Workshop on Multi-Media Database Management Systems (Cat. No.98TB100249).

[42]  A. Lippman,et al.  A Bayesian video modeling framework for shot segmentation and content characterization , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[43]  Paul A. Viola,et al.  Structure Driven Image Database Retrieval , 1997, NIPS.

[44]  Alexander G. Hauptmann,et al.  Informedia: news-on-demand multimedia information acquisition and retrieval , 1997 .

[45]  Tom Minka,et al.  Modeling user subjectivity in image libraries , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[46]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  S. Sclaroff,et al.  Combining textual and visual cues for content-based image retrieval on the World Wide Web , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[48]  Douglas Keislar,et al.  Content-Based Classification, Search, and Retrieval of Audio , 1996, IEEE Multim..