Evaluating multimodal relevance feedback techniques for medical image retrieval

AbstractMedical image retrieval can assist physicians in finding information supporting their diagnosis and fulfilling information needs. Systems that allow searching for medical images need to provide tools for quick and easy navigation and query refinement as the time available for information search is often short. Relevance feedback is a powerful tool in information retrieval. This study evaluates relevance feedback techniques with regard to the content they use. A novel relevance feedback technique that uses both text and visual information of the results is proposed. The two information modalities from the image examples are fused either at the feature level using the Rocchio algorithm or at the query list fusion step using a common late fusion rule. Results using the ImageCLEF 2012 benchmark database for medical image retrieval show the potential of relevance feedback techniques in medical image retrieval. The mean average precision (mAP) is used as the evaluation metric and the proposed method outperforms commonly-used methods. The baseline without feedback reached 16 % whereas the relevance feedback with 20 images reached up to 26.35 % with three steps and when using 100 images up to 34.87 % in four steps. Most improvements occur in the first two steps of relevance feedback and then results start to become relatively flat. This might also be due to only using positive feedback as negative feeback often also improves results after more steps. The effect of relevance feedback in automatically spelling corrected and translated queries is investigated as well. Results without mistakes were better than spell-corrected results but the spelling correction more than double results over non-corrected retrieval. Multimodal relevance feedback has shown to be able to help visual medical information retrieval. Next steps include integrating semantics into relevance feedback techniques to benefit from the structured knowledge of ontologies and experimenting on the fusion of text and visual information.

[1]  Thierry Pun,et al.  Content-based query of image databases: inspirations from text retrieval , 2000, Pattern Recognit. Lett..

[2]  Yue Li,et al.  Mammogram Retrieval: Image Selection Strategy of Relevance Feedback for Locating Similar Lesions , 2011, Int. J. Digit. Libr. Syst..

[3]  Bipin C. Desai,et al.  A Framework for Medical Image Retrieval Using Machine Learning and Statistical Similarity Matching Techniques With Relevance Feedback , 2007, IEEE Transactions on Information Technology in Biomedicine.

[4]  C. Langlotz.  RadLex: a new method for indexing online educational materials. , 2006, Radiographics : a review publication of the Radiological Society of North America, Inc.

[5]  Ingemar J. Cox,et al.  Evaluating Relevance Feedback Algorithms for Searching on Small Displays , 2005, ECIR.

[6]  H Müller,et al.  A Survey on Visual Information Search Behavior and Requirements of Radiologists , 2012, Methods of Information in Medicine.

[7]  Thierry Pun,et al.  Strategies for positive and negative relevance feedback in image retrieval , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[8]  Henning Müller,et al.  Bag-of-Colors for Biomedical Document Image Classification , 2012, MCBR-CDS.

[9]  George R. Thoma,et al.  A Learning-Based Similarity Fusion and Filtering Approach for Biomedical Image Retrieval Using SVM Classification and Relevance Feedback , 2011, IEEE Transactions on Information Technology in Biomedicine.

[10]  Henning Müller,et al.  Overview of the ImageCLEF 2012 Medical Image Retrieval and Classification Tasks , 2012, CLEF.

[11]  Henning Müller,et al.  The medGIFT Group in ImageCLEFmed 2013 , 2013, CLEF.

[12]  Philipp Koehn,et al.  Moses: Open Source Toolkit for Statistical Machine Translation , 2007, ACL.

[13]  Thomas S. Huang,et al.  Relevance feedback techniques in interactive content-based image retrieval , 1997, Electronic Imaging.

[14]  André Mourão,et al.  NovaMedSearch: a multimodal search engine for medical case-based retrieval , 2013, OAIR.

[15]  Wei-Ying Ma,et al.  Alternating Feature Spaces in Relevance Feedback , 2001, MULTIMEDIA '01.

[16]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[17]  N. Boujemaa,et al.  Relevance Feedback for Image Retrieval : a Short Survey , 2004 .

[18]  Mounia Lalmas,et al.  A survey on the use of relevance feedback for information access systems , 2003, The Knowledge Engineering Review.

[19]  Henning Müller,et al.  The ImageCLEF Medical Retrieval Task at ICPR 2010 - Information Fusion to Combine Visual and Textual Information , 2010, ICPR Contests.

[20]  Jing Li,et al.  Relevance Feedback in Content-Based Image Retrieval: A Survey , 2013, Handbook on Neural Information Processing.

[21]  Marco La Cascia,et al.  Image Digestion and Relevance Feedback in the ImageRover WWW Search Engine , 1997 .

[22]  Neill W. Campbell,et al.  Iterative refinement by relevance feedback in content-based digital image retrieval , 1998, MULTIMEDIA '98.

[23]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[24]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[25]  Henning Müller,et al.  The ImageCLEF Medical Retrieval Task at ICPR 2010 – Information Fusion , 2010, 2010 20th International Conference on Pattern Recognition.

[26]  Thierry Pun,et al.  Learning from User Behavior in Image Retrieval: Application of Market Basket Analysis , 2004, International Journal of Computer Vision.

[27]  Henning Müller,et al.  User Tests for Assessing a Medical Image Retrieval System: A Pilot Study , 2013, MedInfo.

[28]  Ingemar J. Cox,et al.  Target testing and the PicHunter Bayesian multimedia retrieval system , 1996, Proceedings of the Third Forum on Research and Technology Advances in Digital Libraries,.

[29]  Henning Müller,et al.  Fusion Techniques for Combining Textual and Visual Information Retrieval , 2010, ImageCLEF.