STRAINet: Spatially Varying sTochastic Residual AdversarIal Networks for MRI Pelvic Organ Segmentation

Accurate segmentation of pelvic organs is important for prostate radiation therapy. Modern radiation therapy starts to use a magnetic resonance image (MRI) as an alternative to computed tomography image because of its superior soft tissue contrast and also free of risk from radiation exposure. However, segmentation of pelvic organs from MRI is a challenging problem due to inconsistent organ appearance across patients and also large intrapatient anatomical variations across treatment days. To address such challenges, we propose a novel deep network architecture, called “Spatially varying sTochastic Residual AdversarIal Network” (STRAINet), to delineate pelvic organs from MRI in an end-to-end fashion. Compared to the traditional fully convolutional networks (FCN), the proposed architecture has two main contributions: 1) inspired by the recent success of residual learning, we propose an evolutionary version of the residual unit, i.e., stochastic residual unit, and use it to the plain convolutional layers in the FCN. We further propose long-range stochastic residual connections to pass features from shallow layers to deep layers; and 2) we propose to integrate three previously proposed network strategies to form a new network for better medical image segmentation: a) we apply dilated convolution in the smallest resolution feature maps, so that we can gain a larger receptive field without overly losing spatial information; b) we propose a spatially varying convolutional layer that adapts convolutional filters to different regions of interest; and c) an adversarial network is proposed to further correct the segmented organ structures. Finally, STRAINet is used to iteratively refine the segmentation probability maps in an autocontext manner. Experimental results show that our STRAINet achieved the state-of-the-art segmentation accuracy. Further analysis also indicates that our proposed network components contribute most to the performance.

[1]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Serge J. Belongie,et al.  Residual Networks Behave Like Ensembles of Relatively Shallow Networks , 2016, NIPS.

[3]  Hao Chen,et al.  3D Fully Convolutional Networks for Intervertebral Disc Localization and Segmentation , 2016, MIAR.

[4]  Philippe Delachartre,et al.  Multi-pass 3 D convolutional neural network segmentation of prostate MRI images , 2017 .

[5]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[6]  C. Davatzikos,et al.  Multi-Atlas Segmentation of the Prostate: A Zooming Process with Robust Registration and Atlas Selection , 2012 .

[7]  Raquel Urtasun,et al.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks , 2016, NIPS.

[8]  Dinggang Shen,et al.  Automated Segmentation of 3D US Prostate Images Using Statistical Texture-Based Matching Method , 2003, MICCAI.

[9]  Yizhou Yu,et al.  Contrast-Oriented Deep Neural Networks for Salient Object Detection , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Hao Chen,et al.  Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images , 2017, AAAI.

[11]  Yaozong Gao,et al.  Fully convolutional networks for multi-modality isointense infant brain image segmentation , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[12]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[13]  Bingbing Ni,et al.  Learning Semantic-Aligned Action Representation , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Klaus H. Maier-Hein,et al.  Adversarial Networks for the Detection of Aggressive Prostate Cancer , 2017, ArXiv.

[15]  C. Fiorino,et al.  Intra- and inter-observer variability in contouring prostate and seminal vesicles: implications for conformal treatment planning. , 1998, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[16]  Hanqing Lu,et al.  Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[18]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Ronald M. Summers,et al.  Active appearance model and deep learning for more accurate prostate segmentation on MRI , 2016, SPIE Medical Imaging.

[20]  Fuyong Xing,et al.  Deep Learning in Microscopy Image Analysis: A Survey. , 2018, IEEE transactions on neural networks and learning systems.

[21]  Dorin Comaniciu,et al.  Marginal Space Learning for Medical Image Analysis: Efficient Detection and Segmentation of Anatomical Structures , 2014 .

[22]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Yaozong Gao,et al.  Deformable MR Prostate Segmentation via Deep Feature Learning and Sparse Patch Matching , 2017, Deep Learning for Medical Image Analysis.

[24]  R. Lenkinski,et al.  Accurate prostate volume estimation using multifeature active shape models on T2-weighted MRI. , 2011, Academic radiology.

[25]  Anant Madabhushi,et al.  Multifeature Landmark-Free Active Appearance Models: Application to Prostate MRI Segmentation , 2012, IEEE Transactions on Medical Imaging.

[26]  Florian Jung,et al.  Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge , 2014, Medical Image Anal..

[27]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Pingkun Yan,et al.  Label Image Constrained Multiatlas Selection , 2015, IEEE Transactions on Cybernetics.

[30]  Qianjin Feng,et al.  Segmenting CT Prostate Images Using Population and Patient-Specific Statistics for Radiotherapy , 2009, ISBI.

[31]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Xingyu Wang,et al.  Sparse Bayesian Classification of EEG for Brain–Computer Interface , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Rainer Lienhart,et al.  An extended set of Haar-like features for rapid object detection , 2002, Proceedings. International Conference on Image Processing.

[34]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[35]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Ian D. Reid,et al.  RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[38]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[39]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[40]  Zhenfeng Zhang,et al.  Superpixel-Based Segmentation for 3D Prostate MR Images , 2016, IEEE Transactions on Medical Imaging.

[41]  Dinggang Shen,et al.  Medical Image Synthesis with Deep Convolutional Adversarial Networks , 2018, IEEE Transactions on Biomedical Engineering.

[42]  Bo Du,et al.  Deeply-supervised CNN for prostate segmentation , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[43]  Ehsan Adeli,et al.  3-D Fully Convolutional Networks for Multimodal Isointense Infant Brain Image Segmentation , 2019, IEEE Transactions on Cybernetics.

[44]  Yinghuan Shi,et al.  Automatic Prostate MR Image Segmentation with Sparse Label Propagation and Domain-Specific Manifold Regularization , 2013, IPMI.

[45]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[46]  Zhuowen Tu,et al.  Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[48]  Mitko Veta,et al.  Adversarial Training and Dilated Convolutions for Brain MRI Segmentation , 2017, DLMIA/ML-CDS@MICCAI.

[49]  Stefan Klein,et al.  Automatic segmentation of the prostate in 3D MR images by atlas matching using localized mutual information. , 2008, Medical physics.

[50]  Kilian Q. Weinberger,et al.  Deep Networks with Stochastic Depth , 2016, ECCV.

[51]  Syed Muhammad Anwar,et al.  Deep Learning in Medical Image Analysis , 2017 .

[52]  Shu Liao,et al.  Representation Learning: A Unified Deep Learning Framework for Automatic Prostate MR Segmentation , 2013, MICCAI.

[53]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[54]  Irina Voiculescu,et al.  An Overview of Current Evaluation Methods Used in Medical Image Segmentation , 2015 .