Content-Based Microscopic Image Retrieval System for Multi-Image Queries

In this paper, we describe the design and development of a multitiered content-based image retrieval (CBIR) system for microscopic images utilizing a reference database that contains images of more than one disease. The proposed CBIR system uses a multitiered approach to classify and retrieve microscopic images involving their specific subtypes, which are mostly difficult to discriminate and classify. This system enables both multi-image query and slide-level image retrieval in order to protect the semantic consistency among the retrieved images. New weighting terms, inspired from information retrieval theory, are defined for multiple-image query and retrieval. The performance of the system was tested on a dataset including 1666 imaged high power fields extracted from 57 follicular lymphoma (FL) tissue slides with three subtypes and 44 neuroblastoma (NB) tissue slides with four subtypes. Each slide is semantically annotated according to their subtypes by expert pathologists. By using leave-one-slide out testing scheme, the multi-image query algorithm with the proposed weighting strategy achieves about 93% and 86% of average classification accuracy at the first rank retrieval, outperforming the image-level retrieval accuracy by about 38 and 26 percentage points, for FL and NB diseases, respectively.

[1]  Karen Spärck Jones.  A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[2]  Carla E. Brodley,et al.  ASSERT: A Physician-in-the-Loop Content-Based Retrieval System for HRCT Image Databases , 1999, Comput. Vis. Image Underst..

[3]  H. Greenspan,et al.  Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. , 2010, Radiology.

[4]  Sonja Zillner,et al.  Semantics and CBIR: a medical imaging perspective , 2008, CIVR '08.

[5]  Joel H. Saltz,et al.  Histopathological Image Analysis Using Model-Based Intermediate Representations and Color Texture: Follicular Lymphoma Grading , 2009, J. Signal Process. Syst..

[6]  Jiebo Luo,et al.  Color object detection using spatial-color joint probability functions , 2004, IEEE Transactions on Image Processing.

[7]  Akiko Aizawa,et al.  An information-theoretic perspective of tf-idf measures , 2003, Inf. Process. Manag..

[8]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[9]  Jake K. Aggarwal,et al.  Feature Integration, Multi-image Queries and Relevance Feedback in Image Retrieval , 2003 .

[10]  Donald Metzler,et al.  Generalized inverse document frequency , 2008, CIKM '08.

[11]  Haralampos Karanikas,et al.  A Pattern Similarity Scheme for Medical Image Retrieval , 2009, IEEE Transactions on Information Technology in Biomedicine.

[12]  Lei Zhang,et al.  Contents lists available at ScienceDirect Pattern Recognition , 2022 .

[13]  Antoine Geissbühler,et al.  A Review of Content{Based Image Retrieval Systems in Medical Applications { Clinical Bene(cid:12)ts and Future Directions , 2022 .

[14]  Hayit Greenspan,et al.  Content-Based Image Retrieval in Radiology: Current Status and Future Directions , 2010, Journal of Digital Imaging.

[15]  Rudolf Hanka,et al.  Histological image retrieval based on semantic content analysis , 2003, IEEE Transactions on Information Technology in Biomedicine.

[16]  Edward Y. Chang,et al.  Why one example is not enough for an image query , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[17]  L. Rodney Long,et al.  SPIRS: A Framework for Content-based Image Retrieval from Large Biomedical Databases , 2007, MedInfo.

[18]  George R. Thoma,et al.  A classification-driven similarity matching framework for retrieval of biomedical images , 2010, MIR '10.

[19]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[20]  Fabio A. González,et al.  Visual Pattern Analysis in Histopathology Images Using Bag of Features , 2009, CIARP.

[21]  Ivan Bajla,et al.  Robust Object Recognition under Partial Occlusions Using NMF , 2008, Comput. Intell. Neurosci..

[22]  Jun Kong,et al.  Computer-aided prognosis of neuroblastoma on whole-slide images: Classification of stromal development , 2009, Pattern Recognit..

[23]  Michael Unser,et al.  Sum and Difference Histograms for Texture Classification , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  R H Choplin,et al.  Picture archiving and communication systems: an overview. , 1992, Radiographics : a review publication of the Radiological Society of North America, Inc.

[25]  Hossein Pourghassem,et al.  Content-based medical image classification using a new hierarchical merging scheme , 2008, Comput. Medical Imaging Graph..

[26]  Vipin Chaudhary,et al.  Content based sub-image retrieval system for high resolution pathology images using salient interest points , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[27]  Joel H. Saltz,et al.  Stroma classification for neuroblastoma on graphics processors , 2009, Int. J. Data Min. Bioinform..

[28]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[30]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[31]  Lin Yang,et al.  PathMiner: A Web-Based Tool for Computer-Assisted Diagnostics in Pathology , 2009, IEEE Transactions on Information Technology in Biomedicine.

[32]  Anant Madabhushi,et al.  A boosted distance metric: application to content based image retrieval and classification of digitized histopathology , 2009, Medical Imaging.

[33]  Metin Nafi Gürcan,et al.  Automatic detection of follicular regions in H&E images using iterative shape index , 2011, Comput. Medical Imaging Graph..

[34]  Bipin C. Desai,et al.  A Framework for Medical Image Retrieval Using Machine Learning and Statistical Similarity Matching Techniques With Relevance Feedback , 2007, IEEE Transactions on Information Technology in Biomedicine.

[35]  Thomas Martin Deserno,et al.  Ontology of Gaps in Content-Based Image Retrieval , 2009, Journal of Digital Imaging.

[36]  Xiang Li,et al.  Content-Based Image Retrieval of Skin Lesions by Evolutionary Feature Synthesis , 2010, EvoApplications.

[37]  Lei Zheng,et al.  Design and analysis of a content-based pathology image retrieval system , 2003, IEEE Transactions on Information Technology in Biomedicine.

[38]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[39]  Rong Jin,et al.  A Boosting Framework for Visuality-Preserving Distance Metric Learning and Its Application to Medical Image Retrieval , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Nam Chul Kim,et al.  Content-Based Image Retrieval Using Multiresolution Color and Texture Features , 2008, IEEE Transactions on Multimedia.

[41]  Charles Elkan,et al.  Deriving TF-IDF as a Fisher Kernel , 2005, SPIRE.

[42]  K K Matthay,et al.  The International Neuroblastoma Pathology Classification (the Shimada system) , 1999, Cancer.