A Multimodal Search Engine for Medical Imaging Studies

The use of digital medical imaging systems in healthcare institutions has increased significantly, and the large amounts of data in these systems have led to the conception of powerful support tools: recent studies on content-based image retrieval (CBIR) and multimodal information retrieval in the field hold great potential in decision support, as well as for addressing multiple challenges in healthcare systems, such as computer-aided diagnosis (CAD). However, the subject is still under heavy research, and very few solutions have become part of Picture Archiving and Communication Systems (PACS) in hospitals and clinics. This paper proposes an extensible platform for multimodal medical image retrieval, integrated in an open-source PACS software with profile-based CBIR capabilities. In this article, we detail a technical approach to the problem by describing its main architecture and each sub-component, as well as the available web interfaces and the multimodal query techniques applied. Finally, we assess our implementation of the engine with computational performance benchmarks.

[1]  Javed A. Aslam,et al.  Relevance score normalization for metasearch , 2001, CIKM '01.

[2]  Charles L. A. Clarke,et al.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods , 2009, SIGIR.

[3]  Allan Hanbury,et al.  KHRESMOI: towards a multi-lingual search and access system for biomedical information , 2011 .

[4]  Thierry Pun,et al.  MRML: an extensible communication protocol for interoperability and benchmarking of multimedia information retrieval systems , 2000, SPIE Optics East.

[5]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[6]  Jong-Hak Lee,et al.  Analyses of multiple evidence combination , 1997, SIGIR '97.

[7]  Georg Langs,et al.  A Visual Information Retrieval System for Radiology Reports and the Medical Literature , 2014, MMM.

[8]  Daekeun You,et al.  Multimodal biomedical image retrieval using hierarchical classification and modality fusion , 2013, International Journal of Multimedia Information Retrieval.

[9]  Degui Xiao,et al.  Medical Image Retrieval: A Multimodal Approach , 2014, Cancer informatics.

[10]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[11]  João Magalhães,et al.  Multimodal medical information retrieval with unsupervised rank fusion , 2015, Comput. Medical Imaging Graph..

[12]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[13]  Henning Müllera,et al.  Health care professionals ’ image use and search behaviour , 2006 .

[14]  Tiago Marques Godinho,et al.  Anatomy of an Extensible Open Source PACS , 2016, Journal of Digital Imaging.

[15]  Georg Langs,et al.  User-oriented evaluation of a medical image retrieval system for radiologists , 2015, Int. J. Medical Informatics.

[16]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[17]  Henning Müller,et al.  Architecture and applications of the Parallel Distributed Image Search Engine (ParaDISE) , 2014, GI-Jahrestagung.

[18]  Rainer Lienhart,et al.  The Holy Grail of Multimedia Information Retrieval: So Close or Yet So Far Away? , 2008 .

[19]  André Mourão,et al.  NovaMedSearch: a multimodal search engine for medical case-based retrieval , 2013, OAIR.

[20]  José Luís Oliveira,et al.  A RESTful Image Gateway for Multiple Medical Image Repositories , 2012, IEEE Transactions on Information Technology in Biomedicine.

[21]  Henning Müller,et al.  Gesture Interaction for Content--based Medical Image Retrieval , 2014, ICMR.

[22]  Mohan S. Kankanhalli,et al.  Multimodal fusion for multimedia analysis: a survey , 2010, Multimedia Systems.

[23]  Daniel L Rubin,et al.  Content-based image retrieval in radiology: analysis of variability in human perception of similarity , 2015, Journal of medical imaging.

[24]  Frederico Valente,et al.  Dicoogle, a Pacs Featuring Profiled Content Based Image Retrieval , 2013, PloS one.

[25]  H Müller.  Medical multimedia retrieval 2.0. , 2008, Yearbook of medical informatics.

[26]  Hayit Greenspan,et al.  Content-Based Image Retrieval in Radiology: Current Status and Future Directions , 2010, Journal of Digital Imaging.

[27]  Henning Müller,et al.  Evaluating multimodal relevance feedback techniques for medical image retrieval , 2015, Information Retrieval Journal.

[28]  José Luís Oliveira,et al.  Indexing and retrieving DICOM data in disperse and unstructured archives , 2008, International Journal of Computer Assisted Radiology and Surgery.