Model Formulation: Advancing Biomedical Image Retrieval: Development and Analysis of a Test Collection

OBJECTIVE Develop and analyze results from an image retrieval test collection. METHODS After participating research groups obtained and assessed results from their systems in the image retrieval task of Cross-Language Evaluation Forum, we assessed the results for common themes and trends. In addition to overall performance, results were analyzed on the basis of topic categories (those most amenable to visual, textual, or mixed approaches) and run categories (those employing queries entered by automated or manual means as well as those using visual, textual, or mixed indexing and retrieval methods). We also assessed results on the different topics and compared the impact of duplicate relevance judgments. RESULTS A total of 13 research groups participated. Analysis was limited to the best run submitted by each group in each run category. The best results were obtained by systems that combined visual and textual methods. There was substantial variation in performance across topics. Systems employing textual methods were more resilient to visually oriented topics than those using visual methods were to textually oriented topics. The primary performance measure of mean average precision (MAP) was not necessarily associated with other measures, including those possibly more pertinent to real users, such as precision at 10 or 30 images. CONCLUSIONS We developed a test collection amenable to assessing visual and textual methods for image retrieval. Future work must focus on how varying topic and run types affect retrieval performance. Users' studies also are necessary to determine the best measures for evaluating the efficacy of image retrieval systems.

[1]  Miguel E. Ruiz,et al.  UB at CLEF 2005: Bilingual CLIR and Medical Image Retrieval Tasks , 2005, CLEF.

[2]  Alan F. Smeaton,et al.  Large Scale Evaluations of Multimedia Information Retrieval: The TRECVid Experience , 2005, CIVR.

[3]  O. Ratib,et al.  Casimage Project: A Digital Teaching Files Authoring Environment , 2004, Journal of thoracic imaging.

[4]  Miguel Ángel García Cumbreras,et al.  The University of Jaén at ImageCLEF 2005: Adhoc and Medical Tasks , 2005, CLEF.

[5]  Gobinda G. Chowdhury,et al.  TREC: Experiment and Evaluation in Information Retrieval , 2007 .

[6]  William R. Hersh,et al.  Manual Query Modification and Data Fusion for Medical Image Retrieval , 2005, CLEF.

[7]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[8]  Hermann Ney,et al.  FIRE in ImageCLEF 2005: Combining Content-based Image Retrieval with Textual Information Retrieval , 2005, CLEF.

[9]  Kristopher N Jones,et al.  Group for research in pathology education online resources to facilitate pathology instruction. , 2002, Archives of pathology & laboratory medicine.

[10]  Ellen M. Voorhees,et al.  Evaluating Evaluation Measure Stability , 2000, SIGIR 2000.

[11]  Karen Spärck Jones.  Reflections on TREC , 1995, Inf. Process. Manag..

[12]  Bipin C. Desai,et al.  Supervised Machine Learning based Medical Image Annotation and Retrieval , 2005, CLEF.

[13]  Daniel Tretter,et al.  A Web-Based Secure System for the Distributed Printing of Documents and Images , 1998, J. Vis. Commun. Image Represent..

[14]  M J Mihatsch,et al.  [Web-based learning tools in pathology]. , 2003, Der Pathologe.

[15]  R A Greenes,et al.  The findings--diagnosis continuum: implications for image descriptions and clinical databases. , 1992, Proceedings. Symposium on Computer Applications in Medical Care.

[16]  Romaric Besançon,et al.  Data Fusion of Retrieval Results from Different Media: Experiments at ImageCLEF 2005 , 2005, CLEF.

[17]  Christian Lovis,et al.  The Use of MedGIFT and EasyIR for ImageCLEF 2005 , 2005, CLEF.

[18]  Antoine Geissbühler,et al.  A Review of Content{Based Image Retrieval Systems in Medical Applications { Clinical Bene(cid:12)ts and Future Directions , 2022 .

[19]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[20]  Patrice Degoulet,et al.  Towards content-based image retrieval in a HIS-integrated PACS , 2000, AMIA.

[21]  Corinne Joergensen.  Retrieving the unretrievable in electronic imaging systems: emotions, themes, and stories , 1999, Electronic Imaging.

[22]  José Luis Martínez-Fernández,et al.  Combining Textual and Visual Features for Image Retrieval , 2005, CLEF.

[23]  Qi Tian,et al.  Combining Visual Features for Medical Image Retrieval and Annotation , 2005, CLEF.

[24]  Carol Peters,et al.  Cross-Language Evaluation Forum: Objectives, Results, Achievements , 2004, Information Retrieval.

[25]  Ellen M. Voorhees,et al.  Retrieval evaluation with incomplete information , 2004, SIGIR '04.

[26]  Joo-Hwee Lim,et al.  A Structured Visual Learning Approach Mixed with Ontology Dimensions for Medical Queries , 2005, CLEF.

[27]  Wei-Pang Yang,et al.  Combining Textual and Visual Features for Cross-Language Medical Image Retrieval , 2005, CLEF.

[28]  Ellen M. Voorhees,et al.  Retrieval System Evaluation , 2005 .

[29]  Thomas Martin Deserno,et al.  The CLEF 2005 Cross-Language Image Retrieval Track , 2003, CLEF.

[30]  Ronald E. Rice,et al.  Influences, usage, and outcomes of Internet health information searching: Multivariate results from the Pew surveys , 2006, Int. J. Medical Informatics.

[31]  Paul Over,et al.  Interactivity at the Text Retrieval Conference (TREC) , 2001, Inf. Process. Manag..

[32]  Thomas Martin Deserno,et al.  Content-Based Retrieval of Medical Images by Combining Global Features , 2005, CLEF.

[33]  S. Uijtdehaage,et al.  Introducing HEAL: The Health Education Assets Library , 2003, Academic medicine : journal of the Association of American Medical Colleges.

[34]  William R Hersh,et al.  Enhancing access to the Bibliome: the TREC 2004 Genomics Track , 2006, Journal of biomedical discovery and collaboration.

[35]  K. Glatz-Krieger,et al.  Webbasierte Lernwerkzeuge für die Pathologie , 2003, Der Pathologe.

[36]  Alexander Horsch,et al.  Establishing an International Reference Image Database for Research and Development in Medical Image Processing , 2004, Methods of Information in Medicine.

[37]  Chris Buckley,et al.  OHSUMED: an interactive retrieval evaluation and new large test collection for research , 1994, SIGIR '94.

[38]  J. Wallis,et al.  An Internet-based nuclear medicine teaching file. , 1995, Journal of nuclear medicine : official publication, Society of Nuclear Medicine.