Evaluation Platform for Content-Based Image Retrieval Systems

In all subfields of information retrieval, test datasets and ground truth data are important tools for testing and comparison of new search methods. This is also reflected by the image retrieval community where several benchmarking activities have been created in past years. However, the number of available test collections is still rather small and the existing ones are often limited in size or accessible only to the participants of benchmarking competitions. In this work, we present a new freely-available large-scale dataset for evaluation of content-based image retrieval systems. The dataset consists of 20 million high-quality images with five visual descriptors and rich and systematic textual annotations, a set of 100 test query objects and a semi-automatically collected ground truth data verified by users. Furthermore, we provide services that enable exploitation and collaborative expansion of the ground truth.

[1]  David Novak,et al.  Generic similarity search engine demonstrated by an image retrieval application , 2009, SIGIR.

[2]  Ramesh C. Jain,et al.  Content without context is meaningless , 2010, ACM Multimedia.

[3]  Leonardo Candela,et al.  Digital Libraries: Research and Development, First International DELOS Conference, Pisa, Italy, February 13-14, 2007, Revised Selected Papers , 2007, DELOS.

[4]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[5]  Mark J. Huiskes,et al.  The MIR flickr retrieval evaluation , 2008, MIR '08.

[6]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[7]  Andrew Trotman,et al.  Comparative Evaluation of XML Information Retrieval Systems: 5th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2006 Dagstuhl Castle, Germany, December 17-20, 2006 Revised and Selected Papers , 2005 .

[8]  Rong Yan,et al.  Semantic concept-based query expansion and re-ranking for multimedia retrieval , 2007, ACM Multimedia.

[9]  Stéphane Marchand-Maillet,et al.  TagCaptcha: annotating images with CAPTCHAs , 2010, ACM Multimedia.

[10]  David Novak,et al.  Building a web-scale image similarity search system , 2010, Multimedia Tools and Applications.

[11]  Adrian Popescu,et al.  Overview of the Wikipedia Retrieval Task at ImageCLEF 2010 , 2010, CLEF.

[12]  Christian Fluhr,et al.  Usage-oriented multimedia information retrieval technological evaluation , 2006, MIR '06.

[13]  Theodora Tsikrika,et al.  The Wikipedia Image Retrieval Task , 2010, ImageCLEF.

[14]  Pavel Zezula,et al.  Similarity Query Postprocessing by Ranking , 2010, Adaptive Multimedia Retrieval.

[15]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[16]  David Novak,et al.  MESSIF: Metric Similarity Search Implementation Framework , 2007, DELOS.

[17]  Thierry Pun,et al.  A Framework for Benchmarking in CBIR , 2003, Multimedia Tools and Applications.

[18]  Marcel Worring,et al.  Benchmarking image and video retrieval: an overview , 2006, MIR '06.

[19]  Thijs Westerveld,et al.  The INEX 2006 Multimedia Track , 2006, INEX.

[20]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[21]  Pavel Zezula,et al.  Combining metric features in large collections , 2008, 2008 IEEE 24th International Conference on Data Engineering Workshop.