Semantic-Enhanced Query Expansion System for Retrieving Medical Image Notes

Most current image retrieval methods require constructing semantic metadata for representing image content. To manually create semantic metadata for medical images is time-consuming, yet it is a crucial component for query expansion. We proposed a new method for searching medical image notes that uses semantic metadata to improve query expansion and leverages a knowledge model developed specifically for the medical image domain to create relevant metadata. We used a syntactic parser and the Unified Medical Language System to analyze the corpus and store text information as semantic metadata in a knowledge model. Our new method has an interactive interface that allows users to provide relevance feedback and construct new queries more efficiently. Sixteen medical professionals evaluated the query expansion module, and each evaluator had prior experience searching for medical images. When using the initial query as the baseline standard, expanded queries achieved a performance boost of 22.6% in terms of the relevance score on first ten results (P-value<0.05). When using Google as another baseline, our system performed 24.6% better in terms of relevance score on the first ten results (P-value<0.05). Overall, 75% of the evaluators said the semantic-enhanced query expansion workflow is logical, easy to follow, and comfortable to use. In addition, 62% of the evaluators preferred using our system instead of Google. Evaluators who were positive about our system found the knowledge map-based visualization of candidate medical search terms helpful in refining cases from the initial search results.

[1]  W. Bruce Croft,et al.  Quary Expansion Using Local and Global Document Analysis , 1996, SIGIR Forum.

[2]  Ryen W. White,et al.  Supporting exploratory search , 2006 .

[3]  Michael Krauthammer,et al.  Yale Image Finder (YIF): a new search engine for retrieving biomedical images , 2008, Bioinform..

[4]  Wolfgang Nejdl,et al.  From keywords to semantic queries - Incremental query construction on the semantic web , 2009, J. Web Semant..

[5]  Hagit Shatkay,et al.  Integrating image data into biomedical text categorization , 2006, ISMB.

[6]  Ryen W. White,et al.  Supporting Exploratory Search, Introduction, Special Issue, Communications of the ACM , 2006 .

[7]  Rong Yan,et al.  Semantic concept-based query expansion and re-ranking for multimedia retrieval , 2007, ACM Multimedia.

[8]  Zhenyu Liu,et al.  Textual Document Indexing and Retrieval via Knowledge Sources and Data Mining , 2002 .

[9]  Jack Minker,et al.  An evaluation of query expansion by the addition of clustered terms for a document retrieval system , 1972, Inf. Storage Retr..

[10]  Tao Chen,et al.  Understanding and classifying image tweets , 2013, ACM Multimedia.

[11]  Michael Isard,et al.  Object retrieval with large vocabularies and fast spatial matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Michael Isard,et al.  Lost in quantization: Improving particular object retrieval in large scale image databases , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  W. Bruce Croft,et al.  Query expansion using local and global document analysis , 1996, SIGIR '96.

[14]  Daniel L. Rubin,et al.  Medical Imaging on the Semantic Web: Annotation and Image Markup , 2008, AAAI Spring Symposium: Semantic Scientific Knowledge Integration.

[15]  Hongfang Liu,et al.  Using data-driven sublanguage pattern mining to induce knowledge models: application in medical image reports knowledge representation , 2018, BMC Medical Informatics and Decision Making.

[16]  Marti A. Hearst.  Clustering versus faceted categories for information exploration , 2006, Commun. ACM.

[17]  Nigel Shadbolt,et al.  NITELIGHT: A Graphical Tool for Semantic Query Construction , 2008 .

[18]  Edward Y. Chang,et al.  Support vector machine active learning for image retrieval , 2001, MULTIMEDIA '01.

[19]  Aviezri S. Fraenkel,et al.  Local Feedback in Full-Text Retrieval Systems , 1977, JACM.

[20]  Stephen E. Robertson,et al.  Selecting good expansion terms for pseudo-relevance feedback , 2008, SIGIR '08.

[21]  Charles E. Kahn,et al.  Automated semantic indexing of figure captions to improve radiology image retrieval. , 2009, Journal of the American Medical Informatics Association : JAMIA.

[22]  Zhenyu Liu,et al.  Knowledge-based query expansion to support scenario-specific retrieval of medical free text , 2005, SAC '05.

[23]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.