Artificial Intelligence based Liver Portal Tract Region Identification and Quantification with Transplant Biopsy Whole-Slide Images

Liver fibrosis staging is clinically important for liver disease progression prediction. As the portal tract fibrotic quantity and size in a liver biopsy correlate with the fibrosis stage, an accurate analysis of portal tract regions is clinically critical. Manual annotations of portal tract regions, however, are time-consuming and subject to large inter- and intra-observer variability. To address such a challenge, we develop a Multiple Up-sampling and Spatial Attention guided UNet model (MUSA-UNet) to segment liver portal tract regions in whole-slide images of liver tissue slides. To enhance the segmentation performance, we propose to use depth-wise separable convolution, the spatial attention mechanism, the residual connection, and multiple up-sampling paths in the developed model. This study includes 53 histopathology whole slide images from patients who received liver transplantation. In total, 6,012 patches derived from 30 images are used for our deep learning model training and validation. The remaining 23 whole slide images are utilized for the model testing. The average liver portal tract segmentation performance of the developed MUSA-UNet is 0.94 (Precision), 0.85 (Recall), 0.89 (F1 Score), 0.89 (Accuracy), 0.80 (Jaccard Index), and 0.91 (Fowlkes–Mallows Index), respectively. The clinical Scheuer fibrosis stage presents a strong correlation with the resulting average portal tract fibrotic area (R=0.681, p<0.001) and portal tract percentage (R=0.335, p=0.02) computed from the MUSA-UNet segmentation results. In conclusion, our developed deep learning model MUSA-UNet can accurately segment portal tract regions from whole-slide images of liver tissue biopsies, presenting its promising potential to assist liver disease diagnosis in a computational manner.

[1]  Y. Ramot,et al.  Microscope-Based Automated Quantification of Liver Fibrosis in Mice Using a Deep Learning Algorithm , 2021, Toxicologic pathology.

[2]  Jerry L Prince,et al.  A Review of Deep Learning in Medical Imaging: Imaging Traits, Technology Trends, Case Studies With Progress Highlights, and Future Promises , 2020, Proceedings of the IEEE.

[3]  Siamak Mehrkanoon,et al.  SmaAt-UNet: Precipitation Nowcasting using a Small Attention-UNet Architecture , 2020, Pattern Recognit. Lett..

[4]  Jun Kong,et al.  Quantitative assessment of liver fibrosis by digital image analysis reveals correlation with qualitative clinical fibrosis staging in liver transplant patients , 2020, PloS one.

[5]  Harvey Lui,et al.  Dense-UNet: a novel multiphoton in vivo cellular image segmentation model based on a convolutional neural network. , 2020, Quantitative imaging in medicine and surgery.

[6]  Ting Zhang,et al.  SD-UNet: Stripping down U-Net for Segmentation of Biomedical Images on Platforms with Low Computational Budgets , 2020, Diagnostics.

[7]  Mohammad Sohel Rahman,et al.  MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation , 2019, Neural Networks.

[8]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[9]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[10]  Won-Chul Bang,et al.  Deep learning with ultrasonography: automated classification of liver fibrosis using a deep convolutional neural network , 2019, European Radiology.

[11]  Geoffrey G. Zhang,et al.  Detection and classification the breast tumors using mask R-CNN on sonograms , 2019, Medicine.

[12]  Jun Kong,et al.  Liver Steatosis Segmentation With Deep Learning Methods , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[13]  Yassine Ruichek,et al.  Survey on semantic segmentation using deep learning techniques , 2019, Neurocomputing.

[14]  Chris Yakopcic,et al.  A State-of-the-Art Survey on Deep Learning Theory and Architectures , 2019, Electronics.

[15]  Albert Montillo,et al.  Deep learning convolutional neural networks for the estimation of liver fibrosis severity from ultrasound texture , 2019, Medical Imaging.

[16]  Yao Lu,et al.  RIC-Unet: An Improved Neural Network Based on Unet for Nuclei Segmentation in Histology Images , 2019, IEEE Access.

[17]  Hanry Yu,et al.  Deep learning enables automated scoring of liver fibrosis stages , 2018, Scientific Reports.

[18]  Jin-Young Choi,et al.  Development and Validation of a Deep Learning System for Staging Liver Fibrosis by Using Contrast Agent-enhanced CT Images in the Liver. , 2018, Radiology.

[19]  Hui Yu,et al.  Segmentation of Lung Nodule in CT Images Based on Mask R-CNN , 2018, 2018 9th International Conference on Awareness Science and Technology (iCAST).

[20]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[21]  Peter H. N. de With,et al.  Stain normalization of histopathology images using generative adversarial networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[22]  O. Abe,et al.  Deep learning for staging liver fibrosis on CT: a pilot study , 2018, European Radiology.

[23]  A. Wee,et al.  Progression and regression of fibrosis in viral hepatitis in the treatment era: the Beijing classification , 2018, Modern Pathology.

[24]  Guang Yang,et al.  A two-stage 3D Unet framework for multi-class segmentation on full resolution image , 2018, ArXiv.

[25]  O. Abe,et al.  Liver Fibrosis: Deep Convolutional Neural Network for Staging by Using Gadoxetic Acid-enhanced Hepatobiliary Phase MR Images. , 2017, Radiology.

[26]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Yu Hu,et al.  Machine-learning-based classification of real-time tissue elastography for hepatic fibrosis in patients with chronic hepatitis B , 2017, Comput. Biol. Medicine.

[28]  Zenghui Wang,et al.  Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review , 2017, Neural Computation.

[29]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.

[30]  David Dagan Feng,et al.  Stacked fully convolutional networks with multi-channel learning: application to medical image segmentation , 2017, The Visual Computer.

[31]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[32]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[34]  Nassir Navab,et al.  Structure-Preserving Color Normalization and Sparse Stain Separation for Histological Images , 2016, IEEE Transactions on Medical Imaging.

[35]  Francesco Visin,et al.  A guide to convolution arithmetic for deep learning , 2016, ArXiv.

[36]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[39]  Nassir Navab,et al.  Structure-preserved color normalization for histological images , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[40]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[41]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Yukako Yagi,et al.  Staining Correction in Digital Pathology by Utilizing a Dye Amount Table , 2015, Journal of Digital Imaging.

[43]  Roy E. Welsch,et al.  Experimenting Liver Fibrosis Diagnostic by Two Photon Excitation Microscopy and Bag-of-Features Image Classification , 2014, Scientific Reports.

[44]  Nico Karssemeijer,et al.  Quantitative analysis of stain variability in histology slides and an algorithm for standardization , 2014, Medical Imaging.

[45]  Hanry Yu,et al.  qFibrosis: a fully-quantitative innovative method incorporating histological features to facilitate accurate fibrosis scoring in animal model and chronic hepatitis B patients. , 2014, Journal of hepatology.

[46]  Yukako Yagi,et al.  Color standardization in whole slide imaging using a color calibration slide , 2014, Journal of pathology informatics.

[47]  May D. Wang,et al.  Comparison of normalization algorithms for cross-batch color segmentation of histopathological images , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[48]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[49]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[50]  R. Standish,et al.  An appraisal of the histopathological assessment of liver fibrosis , 2006, Gut.