IFM3IRS: Information fusion retrieval system with knowledge-assisted text and visual features based on medical conceptual model

The technology of medical data production has been rapidly changed over the past few years. Modern computer technology has created the possibility of creating multi-modal medical images. Medical data often contain multi-modal information such as visual information (image) as well as textual information. Both types of information are important for medical retrieval system (MRS). Due to the information limitation at different levels of sources, the application of information fusion becomes a real need in medical application. In this research, an information fusion framework was built to develop the multi-modality medical image retrieval system (IFM3IRS). The framework utilizes two sources of information involving text and visual-based retrieval process. The application is based on sequential order where the result from text-based process will automatically be the input in visual-based process. The main contributions of this paper are the development of a new ranking model called MedHieCon ranking model which applies semantic concepts of modality, anatomy and pathology in text-based process and also the learning approach of medical images using medical concept model in visual-based process. ImageCLEFmed 2010 data collection was used to evaluate IFM3IRS and it shows that our information fusion framework is in top list among other researchers. Although text-based retrieval system has proven to be a better performance in MRS; it is significant to determine the overall performance improvements which include the fusion of text and image.

[1]  Miguel Ángel García Cumbreras,et al.  SINAI at ImageCLEF 2010 Medical Task , 2010, CLEF.

[2]  W. B. Cavnar,et al.  N-gram-based text categorization , 1994 .

[3]  Richard Chbeir,et al.  System for Medical Image Retrieval: The MIMS Model , 1999, VISUAL.

[4]  Michael Kohnen,et al.  The IRMA code for unique classification of medical images , 2003, SPIE Medical Imaging.

[5]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[6]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[7]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[8]  Henning Müller,et al.  Information Fusion for Combining Visual and Textual Image Retrieval , 2010, 2010 20th International Conference on Pattern Recognition.

[9]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.

[10]  Jimeng Sun,et al.  Medical Case-based Retrieval by Leveraging Medical Ontology and Physician Feedback: UIUC-IBM at ImageCLEF 2010 , 2010, CLEF.

[11]  Henning Müller,et al.  Information Fusion for Combining Visual and Textual Image Retrieval in ImageCLEF@ICPR , 2010, ICPR Contests.

[12]  Cathy H. Wu,et al.  UniProt: the Universal Protein knowledgebase , 2004, Nucleic Acids Res..

[13]  Richard Chbeir,et al.  MIMS: A Prototype for medical image retrieval , 2000, RIAO.

[14]  George R. Thoma,et al.  Biomedical Image Retrieval Using Multimodal Context and Concept Feature Spaces , 2011, MCBR-CDS.

[15]  Arcot Sowmya,et al.  Multi-level classification of emphysema in HRCT lung images , 2009, Pattern Analysis and Applications.

[16]  Tom Ziemke,et al.  On the Definition of Information Fusion as a Field of Research , 2007 .

[17]  Joo-Hwee Lim,et al.  Using Ontology Dimensions and Negative Expansion to solve Precise Queries in the ImageCLEF Medical Task , 2005, CLEF.

[18]  Steven Bedrick,et al.  Improving Retrieval Using External Annotations: OHSU at ImageCLEF 2010 , 2010, CLEF.

[19]  Joo-Hwee Lim,et al.  Using Ontology Dimensions and Negative Expansion to solve Precise Queries in CLEF Medical Task , 2005 .

[20]  Luis Alfonso Ureña López,et al.  Query expansion with a medical ontology to improve a multimodal information retrieval system , 2009, Comput. Biol. Medicine.

[21]  Carol Peters,et al.  Cross-Language Information Retrieval and Evaluation , 2001, Lecture Notes in Computer Science.

[22]  James Llinas,et al.  An introduction to multisensor data fusion , 1997, Proc. IEEE.

[23]  Antoine Geissbühler,et al.  A Review of Content{Based Image Retrieval Systems in Medical Applications { Clinical Bene(cid:12)ts and Future Directions , 2022 .

[24]  Jitendra Paliwal,et al.  Spectral Data Compression and Analyses Techniques to Discriminate Wheat Classes , 2006 .

[25]  Antoine Geissbühler,et al.  Erratum to "A review of content-based image retrieval systems in medical applications - Clinical benefits and future directions" [I. J. Medical Informatics 73 (1) (2004) 1-23] , 2009, Int. J. Medical Informatics.

[26]  Yongyi Yang,et al.  Supervised and unsupervised methods for prostate cancer segmentation with multispectral MRI. , 2010, Medical physics.

[27]  Guandong Xu,et al.  Expanding user’s query with tag-neighbors for effective medical information retrieval , 2014, Multimedia Tools and Applications.

[28]  Henning Müller,et al.  Overview of the CLEF 2009 Medical Image Retrieval Track , 2009, CLEF.

[29]  Ragia Ibrahim Waleed Arafa.  Medical Image Retrieval: ISSR at CLEF 2010 , 2009, CLEF.

[30]  Alexandros Stougiannis,et al.  IPL at ImageCLEF 2010 , 2010, CLEF.

[31]  Henning Müller,et al.  Fusion Techniques for Combining Textual and Visual Information Retrieval , 2010, ImageCLEF.

[32]  Djoerd Hiemstra,et al.  A probabilistic justification for using tf×idf term weighting in information retrieval , 2000, International Journal on Digital Libraries.

[33]  L. Rodney Long,et al.  SPIRS: A Web-based image retrieval system for large biomedical databases , 2009, Int. J. Medical Informatics.

[34]  Gonzalo Navarro,et al.  Merging prediction by partial matching with structural contexts model , 2004, Data Compression Conference, 2004. Proceedings. DCC 2004.

[35]  Olivier Bodenreider,et al.  The Unified Medical Language System (UMLS): integrating biomedical terminology , 2004, Nucleic Acids Res..

[36]  Miguel Ángel García Cumbreras,et al.  University of Jaén at ImageCLEF 2009: Medical and Photo Tasks , 2009, CLEF.

[37]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[38]  Michael Kohnen,et al.  Content-based image retrieval in medical applications for picture archiving and communication systems , 2003, SPIE Medical Imaging.

[39]  Preslav Nakov,et al.  BioText Search Engine: beyond abstract search , 2007, Bioinform..