On the Creation of Query Topics for ImageCLEFphoto

The selection of realistic and representative search requests (or topics) presents one of the most crucial challenges of benchmark creation: not only should these request be representative for the document collection used, but they should also reflect real user information needs, so that the effectiveness measured with the benchmark will correspond to that one might expect to obtain in a practical setting as well. In this paper, we present the methodology we used to develop the query topics for ImageCLEFphoto, a benchmark event for the evaluation of visual information retrieval from generic photographic collections: first, we carry out a log file analysis to establish a pool of realistic and representative topic candidates for the document collection in question. Based on these topic candidates, we then create a set of representative topics against a number of dimensions to provide an element of control of the topic selection and development processes, and we finally discuss the generation and translation of these topics as well as their distribution to the participants of ImageCLEFphoto.

[1]  K. Sparck Jones,et al.  INFORMATION RETRIEVAL TEST COLLECTIONS , 1976 .

[2]  Donna K. Harman,et al.  Overview of the Sixth Text REtrieval Conference (TREC-6) , 1997, Inf. Process. Manag..

[3]  Ellen M. Voorhees,et al.  Overview of the Seventh Text REtrieval Conference , 1998 .

[4]  Ellen M. Voorhees,et al.  The seventh text REtrieval conference (TREC-7) , 1999 .

[5]  Clement H. C. Leung,et al.  Benchmarking for Content-Based Visual Information Search , 2000, VISUAL.

[6]  Christa Womser-Hacker.  Multilingual Topic Generation within the CLEF 2001 Experiments , 2001, CLEF.

[7]  Carol Peters,et al.  European research letter: Cross-language system evaluation: The CLEF campaigns , 2001, J. Assoc. Inf. Sci. Technol..

[8]  Noriko Kando,et al.  Sensitivity of IR systems Evaluation to Topic Difficulty , 2002, LREC.

[9]  Stéphane Marchand-Maillet,et al.  Benchmarking Image Retrieval Applications , 2004 .

[10]  Mark Sanderson,et al.  The CLEF 2004 Cross-Language Image Retrieval Track , 2004, CLEF.

[11]  M. Sanderson,et al.  Analyzing geographic queries , 2004 .

[12]  Thomas Martin Deserno,et al.  The CLEF 2005 Cross-Language Image Retrieval Track , 2003, CLEF.

[13]  Andrew Trotman,et al.  INEX 2005 guidelines for topic development , 2005 .

[14]  Hermann Ney,et al.  FIRE in ImageCLEF 2005: Combining Content-based Image Retrieval with Textual Information Retrieval , 2005, CLEF.

[15]  Paul Clough,et al.  The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information Systems , 2006 .

[16]  Thomas Deselaers,et al.  Overview of the ImageCLEF 2006 Photographic Retrieval and Object Annotation Tasks , 2006, CLEF.

[17]  Wei Vivian Zhang,et al.  Geomodification in Query Rewriting , 2006, GIR.

[18]  Paul Clough,et al.  Using heterogeneous annotation and visual information for the benchmarking of image retrieval systems , 2006, Electronic Imaging.

[19]  Thijs Westerveld,et al.  Benchmarking multimedia search in structured collections , 2006, MIR '06.

[20]  Allan Hanbury,et al.  Overview of the ImageCLEFphoto 2007 Photographic Retrieval Task , 2008, CLEF.

[21]  Michael Grubinger,et al.  Analysis and evaluation of visual information systems performance , 2007 .