Framework for Integration of Medical Image and Text-Based Report Retrieval to Support Radiological Diagnosis

Medical images are vital part of diagnostics and patient treatment. With the advent of technology, there is a rapid increase in the number of radiological images produced every day. Attempts have been made to use a Content Based Image Retrieval (CBIR) approach for assisting in radiological diagnosis. However, this approach suffers from the semantic gap problem. Few text retrieval systems are in place for assisting the radiologist to retrieve similar past cases. However, for the least experienced radiologist it is hard to describe the unknown case using text query. Therefore, the aim of this chapter is integrating the radiological CBIR and text based reports retrieval in order to support radiological diagnosis. The proposed technique is described in three stages: a) retrieval by image similarity, b) retrieval by text, and c) fusion of image and text retrieval for better diagnosis. Number of experiments are demonstrated along with their evaluation techniques on mammogram image database. Framework for Integration of Medical Image and Text-Based Report Retrieval to Support Radiological Diagnosis

[1]  Henning Müller,et al.  Case-based fracture image retrieval , 2012, International Journal of Computer Assisted Radiology and Surgery.

[2]  Noémie Elhadad,et al.  Natural Language Processing in Health Care and Biomedicine , 2014 .

[3]  Raveendran Paramesran,et al.  On the computational aspects of Zernike moments , 2007, Image Vis. Comput..

[4]  Kunio Doi,et al.  Presentation of Similar Images as a Reference for Distinction Between Benign and Malignant Masses on Mammograms: Analysis of Initial Observer Study , 2009, Journal of Digital Imaging.

[5]  Fabio A. González,et al.  Histopathology Image Classification Using Bag of Features and Kernel Functions , 2009, AIME.

[6]  David Page,et al.  Information Extraction for Clinical Data Mining: A Mammography Case Study , 2009, 2009 IEEE International Conference on Data Mining Workshops.

[7]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  C. Friedman.  Semantic Text Parsing for Patient Records , 2005 .

[9]  Rebecca S Lewis,et al.  Disagreement in interpretation: a method for the development of benchmarks for quality assurance in imaging. , 2004, Journal of the American College of Radiology : JACR.

[10]  B. S. Manjunath,et al.  Unsupervised Segmentation of Color-Texture Regions in Images and Video , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Cord Spreckelsen,et al.  Towards case-based medical learning in radiological decision making using content-based image retrieval , 2011, BMC Medical Informatics Decis. Mak..

[12]  Li Lan,et al.  Classification of breast lesions with multimodality computer-aided diagnosis: observer study results on an independent clinical data set. , 2006, Radiology.

[13]  Jessica W T Leung,et al.  Performance benchmarks for diagnostic mammography. , 2005, Radiology.

[14]  Stefan M. Rüger,et al.  Exploring Image, Text and Geographic Evidences in ImageCLEF 2007 , 2007, CLEF.

[15]  Hsinchun Chen,et al.  Exploring the use of concept spaces to improve medical information retrieval , 2000, Decis. Support Syst..

[16]  A. I. Cohn,et al.  Expert system-controlled image display. , 1989, Radiology.

[17]  Jong-Hak Lee,et al.  Analyses of multiple evidence combination , 1997, SIGIR '97.

[18]  Dan Shen,et al.  Performance and Scalability of a Large-Scale N-gram Based Information Retrieval System , 2000, J. Digit. Inf..

[19]  Gabriela Csurka,et al.  XRCE's Participation to ImageCLEFphoto 2007 , 2007, CLEF.

[20]  K. Doi,et al.  Investigation of new psychophysical measures for evaluation of similar images on thoracic computed tomography for distinction between benign and malignant nodules. , 2003, Medical physics.

[21]  Ronald C. Arkin,et al.  Visual interaction: a link between perception and problem solving , 1991, Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics.

[22]  Cordelia Schmid,et al.  Improving Bag-of-Features for Large Scale Image Search , 2010, International Journal of Computer Vision.

[23]  Wei-Pang Yang,et al.  NCTU_DBLAB@ImageCLEFmed 2005: Medical Image Retrieval Task , 2005, CLEF.

[24]  Andrew Stranieri,et al.  Hybrid Technique Based on N-GRAM and Neural Networks for Classification of Mammographic Images , 2014 .

[25]  Hermann Ney,et al.  FIRE in ImageCLEF 2005: Combining Content-based Image Retrieval with Textual Information Retrieval , 2005, CLEF.

[26]  Hermann Ney,et al.  Image Retrieval and Annotation Using Maximum Entropy , 2006, CLEF.

[27]  Lucila Ohno-Machado,et al.  Natural language processing: an introduction , 2011, J. Am. Medical Informatics Assoc..

[28]  Joon Ho Lee,et al.  Combining multiple evidence from different properties of weighting schemes , 1995, SIGIR '95.

[29]  Brijesh Verma,et al.  Fuzzy logic based texture queries for CBIR , 2003, Proceedings Fifth International Conference on Computational Intelligence and Multimedia Applications. ICCIMA 2003.

[30]  Xia Li,et al.  Combination of Radiological and Gray Level Co-occurrence Matrix Textural Features Used to Distinguish Solitary Pulmonary Nodules by Computed Tomography , 2013, Journal of Digital Imaging.

[31]  B. S. Manjunath,et al.  Tools for texture- and color-based search of images , 1997, Electronic Imaging.

[32]  Guojun Lu,et al.  Image indexing and retrieval based on vector quantization , 2007, Pattern Recognit..

[33]  Antoine Geissbühler,et al.  Integrating Content-Based Visual Access Methods into a Medical Case Database , 2003, MIE.

[34]  Marcelo Ossamu Honda,et al.  Implementação e avaliação de um sistema de gerenciamento de imagens médicas com suporte à recuperação baseada em conteúdo , 2008 .

[35]  Efstathios Stamatatos,et al.  Words versus Character n-Grams for Anti-Spam Filtering , 2007, Int. J. Artif. Intell. Tools.

[36]  Payel Ghosh,et al.  Review of medical image retrieval systems and future directions , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[37]  Daniel L. Rubin,et al.  A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations , 2014, J. Biomed. Informatics.

[38]  Halil Kilicoglu,et al.  Word sense disambiguation by selecting the best semantic type based on Journal Descriptor Indexing: Preliminary experiment , 2006 .

[39]  Gabriela Csurka,et al.  Unsupervised Visual and Textual Information Fusion in Multimedia Retrieval - A Graph-based Point of View , 2014, ArXiv.

[40]  Roger Levy,et al.  A new approach to cross-modal multimedia retrieval , 2010, ACM Multimedia.

[41]  Stéfan Jacques Darmoni,et al.  MedIC/CISMeF at ImageCLEF 2006: Image Annotation and Retrieval Tasks , 2006, CLEF.

[42]  Wesley W. Chu,et al.  The phrase-based vector space model for automatic retrieval of free-text medical documents , 2007, Data Knowl. Eng..

[43]  Gabriela Csurka,et al.  Semantic combination of textual and visual information in multimedia retrieval , 2011, ICMR.

[44]  Riccardo Bellazzi,et al.  Supporting decisions in medical applications: the knowledge management perspective , 2002, Int. J. Medical Informatics.

[45]  Andrew Stranieri,et al.  VISUAL CHARACTER N-GRAMS FOR CLASSIFICATION AND RETRIEVAL OF RADIOLOGICAL IMAGES , 2014 .

[46]  Anil K. Jain,et al.  Image retrieval using color and shape , 1996, Pattern Recognit..

[47]  R. Manmatha,et al.  A Model for Learning the Semantics of Pictures , 2003, NIPS.

[48]  K. Doi,et al.  Investigation of psychophysical measure for evaluation of similar images for mammographic masses: Preliminary results. , 2005, Medical physics.

[49]  Philipp Daumke,et al.  Intelligent image retrieval based on radiology reports , 2012, European Radiology.

[50]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[51]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[52]  P L Miller,et al.  Knowledge-based radiologic image retrieval using axes of clinical relevance. , 1990, Computers and biomedical research, an international journal.

[53]  Pierre Tirilly,et al.  Language modeling for bag-of-visual words image categorization , 2008, CIVR '08.

[54]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[55]  Mohan S. Kankanhalli,et al.  Multimodal fusion for multimedia analysis: a survey , 2010, Multimedia Systems.

[56]  Jacques Wainer,et al.  Assessing the Need for Referral in Automatic Diabetic Retinopathy Detection , 2013, IEEE Transactions on Biomedical Engineering.

[57]  J. Lancaster,et al.  Rates of disagreement in imaging interpretation in a group of community hospitals. , 1998, Academic radiology.

[58]  Barbara Caputo,et al.  CLEF2008 Image Annotation Task: an SVM Confidence-Based Approach , 2008, CLEF.

[59]  José Carlos González,et al.  MIRACLE's Combination of Visual and Textual Queries for Medical Image Retrieval , 2005, CLEF.

[60]  H. Greenspan,et al.  Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. , 2010, Radiology.

[61]  Andy Way,et al.  Dublin City University at CLEF 2004: Experiments with the ImageCLEF St. Andrew's Collection , 2004, CLEF.

[62]  Jinchang Ren,et al.  ANN vs. SVM: Which one performs better in classification of MCCs in mammogram imaging , 2012, Knowl. Based Syst..

[63]  Christos Faloutsos,et al.  QBIC project: querying images by content, using color, texture, and shape , 1993, Electronic Imaging.

[64]  Guido C. H. E. de Croon,et al.  Multi-modal Information Retrieval Using FINT , 2004, CLEF.

[65]  H K Huang,et al.  Image-matching as a medical diagnostic support tool (DST) for brain diseases in children. , 2005, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society.

[66]  M. F. Porter,et al.  An algorithm for suffix stripping , 1997 .

[67]  Claudio Carpineto,et al.  A Survey of Automatic Query Expansion in Information Retrieval , 2012, CSUR.

[68]  Meng Wang,et al.  Text Mining in Multimedia , 2012, Mining Text Data.

[69]  Carla E. Brodley,et al.  ASSERT: A PHYSICIAN-IN-THE-LOOP CONTENT-BASED IMAGE RETRIEVAL SYSTEM FOR HRCT IMAGE DATABASES , 1999 .

[70]  Agma J. M. Traina,et al.  MultiWaveMed: a system for medical image retrieval through wavelets transformations , 2003, 16th IEEE Symposium Computer-Based Medical Systems, 2003. Proceedings..

[71]  Antoine Geissbühler,et al.  A Review of Content{Based Image Retrieval Systems in Medical Applications { Clinical Bene(cid:12)ts and Future Directions , 2022 .

[72]  Diana Inkpen,et al.  Clustering for Photo Retrieval at Image CLEF 2008 , 2008, CLEF.

[73]  L. Rodney Long,et al.  Evaluation of shape similarity measurement methods for spine X-ray images , 2004, J. Vis. Commun. Image Represent..

[74]  G. Rubin,et al.  Data explosion: the challenge of multidetector-row CT. , 2000, European journal of radiology.

[75]  Haejun Lee,et al.  Medical Image Retrieval: Past and Present , 2012, Healthcare informatics research.

[76]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[77]  Eric Y. Tao,et al.  Computer-aided, case-based diagnosis of mammographic regions of interest containing microcalcifications. , 2000, Academic radiology.

[78]  Hermann Ney,et al.  Statistical framework for model-based image retrieval in medical applications , 2003, J. Electronic Imaging.

[79]  C. Langlotz.  RadLex: a new method for indexing online educational materials. , 2006, Radiographics : a review publication of the Radiological Society of North America, Inc.

[80]  Christophe Moulin,et al.  UJM at ImageCLEFwiki 2008 , 2008, CLEF.

[81]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[82]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[83]  Muhammad Hussain,et al.  Mass detection in digital mammograms using Gabor filter bank , 2012 .

[84]  Kent A. Spackman,et al.  SNOMED clinical terms: overview of the development process and project status , 2001, AMIA.

[85]  Carla E. Brodley,et al.  ASSERT: A Physician-in-the-Loop Content-Based Retrieval System for HRCT Image Databases , 1999, Comput. Vis. Image Underst..

[86]  Joo-Hwee Lim,et al.  VisMed: A Visual Vocabulary Approach for Medical Image Indexing and Retrieval , 2005, AIRS.

[87]  Michael Isard,et al.  Object retrieval with large vocabularies and fast spatial matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[88]  Agma J. M. Traina,et al.  From Bag-of-Visual-Words to Bag-of-Visual-Phrases Using n-Grams , 2013, 2013 XXVI Conference on Graphics, Patterns and Images.

[89]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[90]  Xiaohui Liu,et al.  Mammogram retrieval on similar mass lesions , 2012, Comput. Methods Programs Biomed..

[91]  Ying Zhang,et al.  Bag-of-Features Based Classification of Breast Parenchymal Tissue in the Mammogram via Jointly Selecting and Weighting Visual Words , 2011, 2011 Sixth International Conference on Image and Graphics.

[92]  L. Rodney Long,et al.  A system for searching uterine cervix images by visual attributes , 2009, 2009 22nd IEEE International Symposium on Computer-Based Medical Systems.

[93]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.

[94]  Amarnath Gupta,et al.  Virage image search engine: an open framework for image management , 1996, Electronic Imaging.

[95]  Henning Müller,et al.  Fusing visual and clinical information for lung tissue classification in high-resolution computed tomography , 2010, Artif. Intell. Medicine.

[96]  Hayit Greenspan,et al.  X-ray Categorization and Retrieval on the Organ and Pathology Level, Using Patch-Based Visual Words , 2011, IEEE Transactions on Medical Imaging.

[97]  José Carlos González,et al.  MIRACLE at ImageCLEFmed 2007: Merging Textual and Visual Strategies to Improve Medical Image Retrieval , 2007, CLEF.

[98]  Gwénolé Quellec,et al.  Wavelet optimization for content-based image retrieval in medical databases , 2010, Medical Image Anal..

[99]  Wen Gao,et al.  Effective and efficient object-based image retrieval using visual phrases , 2006, MM '06.

[100]  Manisha Sharma,et al.  Novel System for Color Logo Recognition Using Optimization and Learning Based Relevance Feedback Technique , 2017, Int. J. Comput. Vis. Image Process..

[101]  Ian W. Ricketts,et al.  The Mammographic Image Analysis Society digital mammogram database , 1994 .

[102]  K. Doi,et al.  Potential usefulness of similar images in the differential diagnosis of clustered microcalcifications on mammograms. , 2009, Radiology.

[103]  Yi-fang Brook Wu,et al.  Identifying important concepts from medical documents , 2006, J. Biomed. Informatics.

[104]  Henning Müller,et al.  Fusion Techniques for Combining Textual and Visual Information Retrieval , 2010, ImageCLEF.

[105]  Thomas Martin Deserno,et al.  Web-based bone age assessment by content-based image retrieval for case-based reasoning , 2012, International Journal of Computer Assisted Radiology and Surgery.

[106]  Berkman Sahiner,et al.  Similarity evaluation in a content-based image retrieval (CBIR) CADx system for characterization of breast masses on ultrasound images. , 2011, Medical physics.

[107]  Hayit Greenspan,et al.  Content-Based Image Retrieval in Radiology: Current Status and Future Directions , 2010, Journal of Digital Imaging.

[108]  Stephen Chu,et al.  Knowledge representation and retrieval using conceptual graphs and free text document self-organisation techniques , 2001, Int. J. Medical Informatics.

[109]  Adil Alpkocak,et al.  DEU at ImageCLEFmed 2009: Evaluating a Re-ranking and Integrated Retrieval Model , 2009, CLEF.

[110]  Wendy W. Chapman,et al.  A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries , 2001, J. Biomed. Informatics.

[111]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[112]  Antoine Geissbühler,et al.  Case-based lung image categorization and retrieval for interstitial lung diseases: clinical workflows , 2011, International Journal of Computer Assisted Radiology and Surgery.

[113]  D. Abraham Chandy,et al.  Texture feature extraction using gray level statistical matrix for content-based mammogram retrieval , 2013, Multimedia Tools and Applications.

[114]  Juan Miguel Ortiz-de-Lazcano-Lobato,et al.  A Growing Neural Gas Approach to Classify Vehicles in Traffic Environments , 2017, Int. J. Comput. Vis. Image Process..

[115]  Henning Müller.  Medical (Visual) Information Retrieval , 2012, PROMISE Winter School.

[116]  Masahiro Endo,et al.  Content-based image-retrieval system in chest computed tomography for a solitary pulmonary nodule: method and preliminary experiments , 2012, International Journal of Computer Assisted Radiology and Surgery.

[117]  John P. Eakins,et al.  Towards intelligent image retrieval , 2002, Pattern Recognit..

[118]  George R. Thoma,et al.  Biomedical CBIR using “bag of keypoints” in a modified inverted index , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[119]  Daniel Racoceanu,et al.  A Semantic Fusion Approach Between Medical Images and Reports Using UMLS , 2006, AIRS.

[120]  Jerry L Prince,et al.  Current methods in medical image segmentation. , 2000, Annual review of biomedical engineering.

[121]  M R Ramaswamy,et al.  MoSearch: a radiologist-friendly tool for finding-based diagnostic report and image retrieval. , 1996, Radiographics : a review publication of the Radiological Society of North America, Inc.

[122]  Haifeng Xu,et al.  Content-based retrieval in picture archiving and communication systems , 2009, Journal of Digital Imaging.

[123]  Daniel Racoceanu,et al.  Fusing Visual and Clinical Information for Lung Tissue Classification in HRCT Data , 2009 .

[124]  Hichem Sahbi,et al.  TELECOMParisTech at ImageClefphoto 2008: Bi-Modal Text and Image Retrieval with Diversity Enhancement , 2008, CLEF.

[125]  Henning Müller,et al.  Overview of the ImageCLEF 2013 Medical Tasks , 2013, CLEF.

[126]  Toufik Taibi.  Design Pattern Formalization Techniques , 2007 .

[127]  Stéphane Marchand-Maillet,et al.  Information Fusion in Multimedia Information Retrieval , 2007, Adaptive Multimedia Retrieval.

[128]  Ana M. García-Serrano,et al.  Experiences at ImageCLEF 2010 using CBIR and TBIR Mixing Information Approaches , 2010, CLEF.

[129]  Clement J. McDonald,et al.  What can natural language processing do for clinical decision support? , 2009, J. Biomed. Informatics.

[130]  José L. V. Mejino,et al.  A reference ontology for biomedical informatics: the Foundational Model of Anatomy , 2003, J. Biomed. Informatics.

[131]  A. Kak,et al.  Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. , 2003, Radiology.

[132]  Yue Li,et al.  Mammogram retrieval through machine learning within BI-RADS standards , 2011, J. Biomed. Informatics.

[133]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[134]  Gabriela Csurka,et al.  Crossing textual and visual content in different application scenarios , 2009, Multimedia Tools and Applications.

[135]  William R. Hersh,et al.  Information Retrieval in Medicine: The SAPHIRE Experience , 1995 .

[136]  Keith J Dreyer,et al.  Informatics in radiology: Render: an online searchable radiology study repository. , 2009, Radiographics : a review publication of the Radiological Society of North America, Inc.

[137]  Fuhui Long,et al.  Fundamentals of Content-Based Image Retrieval , 2003 .

[138]  Frederico Valente,et al.  Dicoogle, a Pacs Featuring Profiled Content Based Image Retrieval , 2013, PloS one.

[139]  Chang-Tsun Li,et al.  A general framework for content-based medical image retrieval with its application to mammograms , 2005, SPIE Medical Imaging.

[140]  Don R. Hush,et al.  Query by image example: The CANDID approach , 1995 .

[141]  Tsunemitsu Horie,et al.  Does image reduction affect the diagnostic accuracy of digital mammograms? , 2013, Medical Imaging.