Deep Learning-Based Retrieval System for Gigapixel Histopathology Cases and the Open Access Literature

Clinical practice is getting increasingly stressful for pathologists due to increasing complexity and time constraints. Histopathology is slowly shifting to digital pathology, thus creating opportunities to allow pathologists to improve reading quality or save time using Artificial Intelligence (AI)-based applications. We aim to enhance the practice of pathologists through a retrieval system that allows them to simplify their workflow, limit the need for second opinions, while also learning in the process. In this work, an innovative retrieval system for digital pathology is integrated within a Whole Slide Image (WSI) viewer, allowing to define regions of interest in images as queries for finding visually similar areas using deep representations. The back-end similarity computation algorithms are based on a multimodal approach, allowing to exploit both text information and content-based image features. Shallow and deep representations of the images were evaluated, the later showed a better overall retrieval performance in a set of 112 whole slide images from biopsies. The system was also tested by pathologists, highlighting its capabilities and suggesting possible ways to improve it and make it more usable in clinical practice. The retrieval system developed can enhance the practice of pathologists by enabling them to use their experience and knowledge to properly control artificial intelligence tools for navigating repositories of images for decision support purposes.

[1]  Allan Hanbury,et al.  Overview of the VISCERAL Retrieval Benchmark 2015 , 2015, MRDM@ECIR.

[2]  Gilles Louppe,et al.  Collaborative analysis of multi-gigapixel imaging data using Cytomine , 2016, Bioinform..

[3]  Wei Liu,et al.  Towards Large-Scale Histopathological Image Analysis: Hashing-Based Image Retrieval , 2015, IEEE Transactions on Medical Imaging.

[4]  Sameer Antani,et al.  Creating a classification of image types in the medical literature for visual categorization , 2012, Other Conferences.

[5]  B. van Ginneken,et al.  Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis , 2016, Scientific Reports.

[6]  Yukako Yagi,et al.  Use of whole slide imaging in surgical pathology quality assurance: design and pilot validation studies. , 2006, Human pathology.

[7]  Daisuke Komura,et al.  Luigi: Large-scale histopathological image retrieval system using deep texture representations , 2018, bioRxiv.

[8]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yu Zhou,et al.  An Image Analysis Resource for Cancer Research: PIIP-Pathology Image Informatics Platform for Visualization, Analysis, and Management. , 2017, Cancer research.

[10]  Ullrich Köthe,et al.  Ilastik: Interactive learning and segmentation toolkit , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[11]  Manfredo Atzori,et al.  Chapter 10 – Analysis of Histopathology Images: From Traditional Machine Learning to Deep Learning , 2017 .

[12]  Brett Delahunt,et al.  Gleason grading: past, present and future , 2012, Histopathology.

[13]  Anne E Carpenter,et al.  CellProfiler 3.0: Next-generation image processing for biology , 2018, PLoS biology.

[14]  Fabio A. González,et al.  A Deep Learning Architecture for Image Representation, Visual Interpretability and Automated Basal-Cell Carcinoma Cancer Detection , 2013, MICCAI.

[15]  C. Cheung,et al.  Modeling complexity in pathologist workload measurement: the Automatable Activity-Based Approach to Complexity Unit Scoring (AABACUS) , 2015, Modern Pathology.

[16]  L. Egevad,et al.  The 2005 International Society of Urological Pathology (ISUP) Consensus Conference on Gleason Grading of Prostatic Carcinoma , 2005, The American journal of surgical pathology.

[17]  Aleksey Boyko,et al.  Detecting Cancer Metastases on Gigapixel Pathology Images , 2017, ArXiv.

[18]  Henning Müller,et al.  A Modern Web Interface for Medical Image Retrieval , 2014 .

[19]  Nicolas Chenouard,et al.  Icy: an open bioimage informatics platform for extended reproducible research , 2012, Nature Methods.

[20]  Manfredo Atzori,et al.  Image Magnification Regression Using DenseNet for Exploiting Histopathology Open Access Content , 2018, COMPAY/OMIA@MICCAI.

[21]  Yiannis S. Boutalis,et al.  CEDD: Color and Edge Directivity Descriptor: A Compact Descriptor for Image Indexing and Retrieval , 2008, ICVS.

[22]  Florian Jug,et al.  Bioimage Informatics in the context of Drosophila research. , 2014, Methods.

[23]  Anne E Carpenter,et al.  CellProfiler: image analysis software for identifying and quantifying cell phenotypes , 2006, Genome Biology.

[24]  Liron Pantanowitz,et al.  Routine Digital Pathology Workflow: The Catania Experience , 2017, Journal of pathology informatics.

[25]  Stephan Saalfeld,et al.  CATMAID: collaborative annotation toolkit for massive amounts of image data , 2009, Bioinform..

[26]  A. Madabhushi,et al.  Histopathological Image Analysis: A Review , 2009, IEEE Reviews in Biomedical Engineering.

[27]  Johannes E. Schindelin,et al.  Fiji: an open-source platform for biological-image analysis , 2012, Nature Methods.

[28]  Tobias Pietzsch,et al.  BigDataViewer: visualization and processing for large image data sets , 2015, Nature Methods.

[29]  Kristine A. Erps,et al.  Overview of telepathology, virtual microscopy, and whole slide imaging: prospects for the future. , 2009, Human pathology.

[30]  Manfredo Atzori,et al.  Deep Multimodal Case-Based Retrieval for Large Histopathology Datasets , 2017, Patch-MI@MICCAI.

[31]  Ambuj K. Singh,et al.  Bisque: a platform for bioimage analysis and management , 2009, Bioinform..

[32]  Mats Andersson,et al.  Convolutional neural networks for an automatic classification of prostate tissue slides with high-grade Gleason score , 2017, Medical Imaging.

[33]  Wei Liu,et al.  Mining histopathological images via hashing-based scalable image retrieval , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[34]  Andrew A. Renshaw,et al.  Rubin??s Pathology. Clinicopathologic Foundations of Medicine , 2008 .

[35]  Fabio A. González,et al.  Histology image search using multimodal fusion , 2014, J. Biomed. Informatics.

[36]  Lin Yang,et al.  Content-based histopathology image retrieval using CometCloud , 2014, BMC Bioinformatics.

[37]  Andrew Zisserman,et al.  Incremental learning of object detectors using a visual shape alphabet , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[38]  Henning Müller,et al.  The Parallel Distributed Image Search Engine (ParaDISE) , 2017, ArXiv.