Using Concept Hierarchies in Text-Based Image Retrieval: A User Evaluation

This paper describes our results from the image retrieval task of iCLEF 2005 based on a comparative user evaluation of two interfaces: one displaying search results as a list; the other organising retrieved images into a hierarchy of concepts displayed on the interface as an interactive menu. Based on a known-item retrieval task, data was analysed with respect to effectiveness, efficiency and user satisfaction. Effectiveness and efficiency were calculated at both the set cut-off time of 5 minutes, and the time after finding the target image (final time). Results showed the list was marginally more effective than the menu at 5 minutes, but the two were equal at final time indicating the menu requires more time to be used effectively. The list was more efficient at both 5 minutes and final time (difference not statistically significant) and users preferred using the menu indicating this could be a potentially interesting and engaging feature for image retrieval.

[1]  Daniela Petrelli,et al.  Which user interaction for cross-language information retrieval? Design issues and reflections , 2006 .

[2]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[3]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[4]  Mark Sanderson,et al.  A Study of User Interaction with a Concept-Based Interactive Query Expansion Support Tool , 2004, ECIR.

[5]  Gerrit C. van der Veer,et al.  Breaking Down Usability , 1999, INTERACT.

[6]  W. Bruce Croft,et al.  Deriving concept hierarchies from text , 1999, SIGIR '99.

[7]  Jacques Savoy,et al.  Selection and Merging Strategies for Multilingual Information Retrieval , 2004, CLEF.

[8]  Daniela Petrelli,et al.  Observing users, designing clarity: A case study on the user-centered design of a cross-language information retrieval system , 2004, J. Assoc. Inf. Sci. Technol..

[9]  Fredric C. Gey,et al.  Accessing Multilingual Information Repositories, 6th Workshop of the Cross-Language Evalution Forum, CLEF 2005, Vienna, Austria, 21-23 September, 2005, Revised Selected Papers , 2006, CLEF.

[10]  Carol Peters,et al.  Multilingual Information Access for Text, Speech and Images, 5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004, Bath, UK, September 15-17, 2004, Revised Selected Papers , 2005, CLEF.

[11]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[12]  Julio Gonzalo,et al.  Overview of the CLEF 2005 Interactive Track , 2005, CLEF.