An Effective MR-Guided CT Network Training for Segmenting Prostate in CT Images

Segmentation of prostate in medical imaging data (e.g., CT, MRI, TRUS) is often considered as a critical yet challenging task for radiotherapy treatment. It is relatively easier to segment prostate from MR images than from CT images, due to better soft tissue contrast of the MR images. For segmenting prostate from CT images, most previous methods mainly used CT alone, and thus their performances are often limited by low tissue contrast in the CT images. In this article, we explore the possibility of using indirect guidance from MR images for improving prostate segmentation in the CT images. In particular, we propose a novel deep transfer learning approach, i.e., MR-guided CT network training (namely MICS-NET), which can employ MR images to help better learning of features in CT images for prostate segmentation. In MICS-NET, the guidance from MRI consists of two steps: (1) learning informative and transferable features from MRI and then transferring them to CT images in a cascade manner, and (2) adaptively transferring the prostate likelihood of MRI model (i.e., well-trained convnet by purely using MR images) with a view consistency constraint. To illustrate the effectiveness of our approach, we evaluate MICS-NET on a real CT prostate image set, with the manual delineations available as the ground truth for evaluation. Our methods generate promising segmentation results which achieve (1) six percentages higher Dice Ratio than the CT model purely using CT images and (2) comparable performance with the MRI model purely using MR images.

[1]  Stephen M. Smith,et al.  A global optimisation method for robust affine registration of brain images , 2001, Medical Image Anal..

[2]  Shouliang Qi,et al.  CNN models discriminating between pulmonary micro-nodules and non-nodules from CT images , 2018, Biomedical engineering online.

[3]  Hayit Greenspan,et al.  Longitudinal Multiple Sclerosis Lesion Segmentation Using Multi-view Convolutional Neural Networks , 2016, LABELS/DLMIA@MICCAI.

[4]  Hao Chen,et al.  Iterative Multi-domain Regularized Deep Learning for Anatomical Structure Detection and Segmentation from Ultrasound Images , 2016, MICCAI.

[5]  Marios Anthimopoulos,et al.  Multi-source Transfer Learning with Convolutional Neural Networks for Lung Pattern Analysis , 2016, IEEE journal of biomedical and health informatics.

[6]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[7]  Tzu-Tsung Wong,et al.  Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation , 2015, Pattern Recognit..

[8]  N. Karssemeijer,et al.  Discriminating solitary cysts from soft tissue lesions in mammography using a pretrained deep convolutional neural network , 2017, Medical physics.

[9]  Shihui Ying,et al.  Multimodal Neuroimaging Feature Learning With Multimodal Stacked Deep Polynomial Networks for Diagnosis of Alzheimer's Disease , 2018, IEEE Journal of Biomedical and Health Informatics.

[10]  Thomas Hofmann,et al.  Analysis of Representations for Domain Adaptation , 2007 .

[11]  Indrin J Chetty,et al.  Automatic Segmentation of the Prostate on CT Images Using Deep Neural Networks (DNN). , 2019, International journal of radiation oncology, biology, physics.

[12]  Shuiwang Ji,et al.  Deep convolutional neural networks for multi-modality isointense infant brain image segmentation , 2015, NeuroImage.

[13]  Dacheng Tao,et al.  A Survey on Multi-view Learning , 2013, ArXiv.

[14]  Dinggang Shen,et al.  MRI‐based prostate cancer detection with high‐level representation and hierarchical classification , 2017, Medical physics.

[15]  Bjoern H. Menze,et al.  Local Structure Prediction with Convolutional Neural Networks for Multimodal Brain Tumor Segmentation , 2015, MCV@MICCAI.

[16]  S. Joshi,et al.  Automatic Segmentation of Intra-treatment CT Images for Adaptive Radiation Therapy of the Prostate , 2005, MICCAI.

[17]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[18]  Yaozong Gao,et al.  Fully convolutional networks for multi-modality isointense infant brain image segmentation , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[19]  Rongrong Guo,et al.  A combined learning algorithm for prostate segmentation on 3D CT images , 2017, Medical physics.

[20]  Rongrong Guo,et al.  A semiautomatic segmentation method for prostate in CT images using local texture classification and statistical shape modeling , 2018, Medical physics.

[21]  Brian B. Avants,et al.  Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain , 2008, Medical Image Anal..

[22]  Yinghuan Shi,et al.  A learning-based CT prostate segmentation method via joint transductive feature selection and regression , 2016, Neurocomputing.

[23]  Chuan Zhou,et al.  Deep Learning in Medical Image Analysis. , 2017, Advances in experimental medicine and biology.

[24]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[25]  Yinghuan Shi,et al.  Prostate Segmentation in CT Images via Spatial-Constrained Transductive Lasso , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Steve B. Jiang,et al.  Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning , 2018, Biomedical Physics & Engineering Express.

[27]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[28]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[29]  Desire Sidibé,et al.  A survey of prostate segmentation methodologies in ultrasound, magnetic resonance and computed tomography images , 2012, Comput. Methods Programs Biomed..

[30]  Pascal Haigron,et al.  Segmentation of pelvic structures for planning CT using a geometrical shape model tuned by a multi-scale edge detector , 2014, Physics in medicine and biology.

[31]  Shu Liao,et al.  Sparse Patch-Based Label Propagation for Accurate Prostate Localization in CT Images , 2013, IEEE Transactions on Medical Imaging.

[32]  Yinghuan Shi,et al.  Semi-Automatic Segmentation of Prostate in CT Images via Coupled Feature Representation and Spatial-Constrained Transductive Lasso , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Siqi Chen,et al.  Segmenting the prostate and rectum in CT imagery using anatomical constraints , 2011, Medical Image Anal..

[34]  Tao Xu,et al.  Multimodal Deep Learning for Cervical Dysplasia Diagnosis , 2016, MICCAI.

[35]  Yinghuan Shi,et al.  Leveraging Coupled Interaction for Multimodal Alzheimer’s Disease Diagnosis , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[36]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[37]  Chunming Li,et al.  Distance Regularized Level Set Evolution and Its Application to Image Segmentation , 2010, IEEE Transactions on Image Processing.

[38]  Subhashini Venugopalan,et al.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. , 2016, JAMA.

[39]  Yinghuan Shi,et al.  MRM-Lasso: A Sparse Multiview Feature Selection Method via Low-Rank Analysis , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[40]  Dinggang Shen,et al.  Segmenting CT prostate images using population and patient-specific statistics for radiotherapy , 2010, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[41]  Luis Álvarez,et al.  A Morphological Approach to Curvature-Based Evolution of Curves and Surfaces , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Chao Lu,et al.  Precise Segmentation of Multiple Organs in CT Volumes Using Learning-Based Approach and Information Theory , 2012, MICCAI.

[43]  Marius Staring,et al.  Adversarial optimization for joint registration and segmentation in prostate CT radiotherapy , 2019, MICCAI.

[44]  Sham M. Kakade,et al.  Multi-view clustering via canonical correlation analysis , 2009, ICML '09.

[45]  Pingkun Yan,et al.  Label Image Constrained Multiatlas Selection , 2015, IEEE Transactions on Cybernetics.

[46]  Tom Vercauteren,et al.  Diffeomorphic demons: Efficient non-parametric image registration , 2009, NeuroImage.

[47]  Dinggang Shen,et al.  Automated Segmentation of 3D US Prostate Images Using Statistical Texture-Based Matching Method , 2003, MICCAI.

[48]  Dinggang Shen,et al.  Optimized prostate biopsy via a statistical atlas of cancer spatial distribution , 2004, Medical Image Anal..

[49]  Dinggang Shen,et al.  3D Deep Learning for Multi-modal Imaging-Guided Survival Time Prediction of Brain Tumor Patients , 2016, MICCAI.

[50]  Yinghuan Shi,et al.  Incomplete-Data Oriented Multiview Dimension Reduction via Sparse Low-Rank Representation , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[51]  Dinggang Shen,et al.  CLASSIC: Consistent Longitudinal Alignment and Segmentation for Serial Image Computing , 2006, NeuroImage.

[52]  Transductive Prostate Segmentation for CT Image Guided Radiotherapy , 2012, MLMI.

[53]  Dinggang Shen,et al.  Learning-based deformable registration of MR brain images , 2006, IEEE Transactions on Medical Imaging.

[54]  Seong-Whan Lee,et al.  Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis , 2014, NeuroImage.

[55]  Yaozong Gao,et al.  Deformable MR Prostate Segmentation via Deep Feature Learning and Sparse Patch Matching , 2016, IEEE Transactions on Medical Imaging.

[56]  Dimitris N. Metaxas,et al.  3D Meshless Prostate Segmentation and Registration in Image Guided Radiotherapy , 2009, MICCAI.

[57]  Shuo Li,et al.  Multi-modal vertebrae recognition using Transformed Deep Convolution Network , 2016, Comput. Medical Imaging Graph..

[58]  Yaozong Gao,et al.  CT male pelvic organ segmentation using fully convolutional networks with boundary sensitive representation , 2019, Medical Image Anal..

[59]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[60]  Bo Du,et al.  Boundary-Weighted Domain Adaptive Neural Network for Prostate MR Image Segmentation , 2019, IEEE Transactions on Medical Imaging.

[61]  Yinghuan Shi,et al.  Does Manual Delineation only Provide the Side Information in CT Prostate Segmentation? , 2017, MICCAI.