Transformer based multiple instance learning for weakly supervised histopathology image segmentation

. Hispathological image segmentation algorithms play a crit-ical role in computer aided diagnosis technology. The development of weakly supervised segmentation algorithm alleviates the problem of medical image annotation that it is time-consuming and labor-intensive. As a subset of weakly supervised learning, Multiple Instance Learning (MIL) has been proven to be effective in segmentation. However, there is a lack of related information between instances in MIL, which limits the further improvement of segmentation performance. In this paper, we propose a novel weakly supervised method for pixel-level segmentation in histopathology images, which introduces Transformer into the MIL framework to capture global or long-range dependencies. The multi-head self-attention in the Transformer establishes the relationship between instances, which solves the shortcoming that instances are independent of each other in MIL. In addition, deep supervision is introduced to over-come the limitation of annotations in weakly supervised methods and make the better utilization of hierarchical information. The state-of-the-art results on the colon cancer dataset demonstrate the superiority of the proposed method compared with other weakly supervised methods. It is worth believing that there is a potential of our approach for various applications in medical images.

[1]  Ming-Ming Cheng,et al.  Online Attention Accumulation for Weakly Supervised Semantic Segmentation , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Xiangyang Ji,et al.  TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classication , 2021, NeurIPS.

[3]  Yan Wang,et al.  TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation , 2021, ArXiv.

[4]  Junzhou Huang,et al.  DT-MIL: Deformable Transformer for Multi-instance Learning on Histopathological Image , 2021, MICCAI.

[5]  Qi Bi,et al.  MIL-VT: Multiple Instance Learning Enhanced Vision Transformer for Fundus Image Classification , 2021, MICCAI.

[6]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[7]  Alina Zare,et al.  Weakly Supervised Minirhizotron Image Segmentation with MIL-CAM , 2020, ECCV Workshops.

[8]  Jitendra Jonnagaddala,et al.  Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks , 2020, Medical Image Anal..

[9]  I. Takeuchi,et al.  Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Dimitris N. Metaxas,et al.  Multi-scale Cell Instance Segmentation with Keypoint Graph based Bounding Boxes , 2019, MICCAI.

[11]  Yunchao Wei,et al.  CCNet: Criss-Cross Attention for Semantic Segmentation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Yunchao Wei,et al.  Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Qiang Qiu,et al.  Weakly Supervised Instance Segmentation Using Class Peak Response , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Max Welling,et al.  Attention-based Deep Multiple Instance Learning , 2018, ICML.

[15]  Zhi-Hua Zhou,et al.  A brief introduction to weakly supervised learning , 2018 .

[16]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[17]  Zhipeng Jia,et al.  Constrained Deep Weak Supervision for Histopathology Image Segmentation , 2017, IEEE Transactions on Medical Imaging.

[18]  Lin Yang,et al.  Transfer Shape Modeling Towards High-Throughput Microscopy Image Segmentation , 2016, MICCAI.

[19]  Hao Chen,et al.  DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Diogo Almeida,et al.  Resnet in Resnet: Generalizing Residual Architectures , 2016, ArXiv.

[21]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[23]  Zhuowen Tu,et al.  Deeply-Supervised Nets , 2014, AISTATS.

[24]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[25]  Zhuowen Tu,et al.  Weakly supervised histopathology cancer image segmentation and classification , 2014, Medical Image Anal..

[26]  Zhuowen Tu,et al.  Multiple clustered instance learning for histopathology cancer image classification, segmentation and clustering , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[28]  Paul A. Viola,et al.  Multiple Instance Boosting for Object Detection , 2005, NIPS.

[29]  Thomas G. Dietterich,et al.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles , 1997, Artif. Intell..