High-resolution 3D abdominal segmentation with random patch network fusion

Deep learning for three dimensional (3D) abdominal organ segmentation on high-resolution computed tomography (CT) is a challenging topic, in part due to the limited memory provide by graphics processing units (GPU) and large number of parameters and in 3D fully convolutional networks (FCN). Two prevalent strategies, lower resolution with wider field of view and higher resolution with limited field of view, have been explored but have been presented with varying degrees of success. In this paper, we propose a novel patch-based network with random spatial initialization and statistical fusion on overlapping regions of interest (ROIs). We evaluate the proposed approach using three datasets consisting of 260 subjects with varying numbers of manual labels. Compared with the canonical "coarse-to-fine" baseline methods, the proposed method increases the performance on multi-organ segmentation from 0.799 to 0.856 in terms of mean DSC score (p-value < 0.01 with paired t-test). The effect of different numbers of patches is evaluated by increasing the depth of coverage (expected number of patches evaluated per voxel). In addition, our method outperforms other state-of-the-art methods in abdominal organ segmentation. In conclusion, the approach provides a memory-conservative framework to enable 3D segmentation on high-resolution CT. The approach is compatible with many base network structures, without substantially increasing the complexity during inference. Given a CT scan with at high resolution, a low-res section (left panel) is trained with multi-channel segmentation. The low-res part contains down-sampling and normalization in order to preserve the complete spatial information. Interpolation and random patch sampling (mid panel) is employed to collect patches. The high-dimensional probability maps are acquired (right panel) from integration of all patches on field of views.

[1]  Dong Yang,et al.  3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training , 2018, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[2]  Bennett A Landman,et al.  Non-local statistical label fusion for multi-atlas segmentation , 2013, Medical Image Anal..

[3]  Bennett A Landman,et al.  Improving Spleen Volume Estimation Via Computer-assisted Segmentation on Clinically Acquired CT Scans. , 2016, Academic radiology.

[4]  Pierrick Coupé,et al.  AssemblyNet: A Novel Deep Decision-Making Process for Whole Brain MRI Segmentation , 2019, MICCAI.

[5]  Fred A. Hamprecht,et al.  Multi-modal Brain Tumor Segmentation using Deep Convolutional Neural Networks , 2014 .

[6]  Lin Yang,et al.  Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Chi-Wing Fu,et al.  H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes , 2018, IEEE Transactions on Medical Imaging.

[8]  Yan Wang,et al.  Abdominal multi-organ segmentation with organ-attention networks and statistical fusion , 2018, Medical Image Anal..

[9]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[10]  Daniel Rueckert,et al.  A Probabilistic Patch-Based Label Fusion Model for Multi-Atlas Segmentation With Registration Refinement: Application to Cardiac MR Images , 2013, IEEE Transactions on Medical Imaging.

[11]  D. Louis Collins,et al.  Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation , 2011, NeuroImage.

[12]  Xinlei Chen,et al.  Prior-Aware Neural Network for Partially-Supervised Multi-Organ Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Jiang Hsieh,et al.  Computed Tomography: Principles, Design, Artifacts, and Recent Advances, Fourth Edition , 2022 .

[14]  Bennett A Landman,et al.  Efficient multi-atlas abdominal segmentation on clinically acquired CT with SIMPLE context learning , 2015, Medical Image Anal..

[15]  Bennett A. Landman,et al.  Hierarchical performance estimation in the statistical label fusion framework , 2014, Medical Image Anal..

[16]  Yi Yu,et al.  Dense U-net Based on Patch-Based Learning for Retinal Vessel Segmentation , 2019, Entropy.

[17]  Yan Wang,et al.  A Fixed-Point Model for Pancreas Segmentation in Abdominal CT Scans , 2016, MICCAI.

[18]  Bo Li,et al.  Shape-constrained multi-atlas segmentation of spleen in CT , 2014, Medical Imaging.

[19]  Yuankai Huo,et al.  Multi-atlas learner fusion: An efficient segmentation approach for large-scale data , 2015, Medical Image Anal..

[20]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[21]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[22]  Yuankai Huo,et al.  Improving splenomegaly segmentation by learning from heterogeneous multi-source labels , 2019, Medical Imaging: Image Processing.

[23]  Daniel Rueckert,et al.  Automated Abdominal Multi-Organ Segmentation With Subject-Specific Atlas Generation , 2013, IEEE Transactions on Medical Imaging.

[24]  Ahmed Bouridane,et al.  Random sampling for patch-based face recognition , 2017, 2017 5th International Workshop on Biometrics and Forensics (IWBF).

[25]  Shunxing Bao,et al.  3D whole brain segmentation using spatially localized atlas network tiles , 2019, NeuroImage.

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Georg Langs,et al.  f‐AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks , 2019, Medical Image Anal..

[28]  Yuichiro Hayashi,et al.  Hierarchical 3D fully convolutional networks for multi-organ segmentation , 2017, ArXiv.

[29]  Klaus H. Maier-Hein,et al.  nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation , 2018, Bildverarbeitung für die Medizin.

[30]  Yuichiro Hayashi,et al.  A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation , 2018, MICCAI.

[31]  Qingmin Liao,et al.  Discriminative patch-based sparse representation for face recognition , 2016, 2016 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC).

[32]  Giovanni Montana,et al.  Deep neural networks for anatomical brain segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Ronald M. Summers,et al.  Unsupervised body part regression via spatially self-ordering convolutional neural networks , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[34]  Alan L. Yuille,et al.  A 3D Coarse-to-Fine Framework for Volumetric Medical Image Segmentation , 2017, 2018 International Conference on 3D Vision (3DV).

[35]  Daniel Rueckert,et al.  Multi-atlas based segmentation of brain images: Atlas selection and its effect on accuracy , 2009, NeuroImage.

[36]  Max A. Viergever,et al.  Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalities , 2016, MICCAI.

[37]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[38]  Georg Langs,et al.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery , 2017, IPMI.

[39]  Tianwei Ni,et al.  Elastic Boundary Projection for 3D Medical Image Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[41]  Hanqiang Cao,et al.  Nearest Neighbor Value Interpolation , 2012, ArXiv.

[42]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[43]  Klaus H. Maier-Hein,et al.  Abstract: nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation , 2019, Bildverarbeitung für die Medizin.

[44]  Daoqiang Zhang,et al.  Sparse Patch-Based Label Fusion for Multi-Atlas Segmentation , 2012, MBIA.

[45]  Yuichiro Hayashi,et al.  An application of cascaded 3D fully convolutional networks for medical image segmentation , 2018, Comput. Medical Imaging Graph..

[46]  Yuankai Huo,et al.  Multi-atlas spleen segmentation on CT using adaptive context learning , 2017, Medical Imaging.

[47]  Sébastien Ourselin,et al.  Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations , 2017, DLMIA/ML-CDS@MICCAI.

[48]  Daguang Xu,et al.  3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes , 2017, MICCAI.

[49]  Paul A. Yushkevich,et al.  Multi-atlas segmentation with joint label fusion and corrective learning—an open source implementation , 2013, Front. Neuroinform..

[50]  Konstantinos Kamnitsas,et al.  Multi-scale 3D convolutional neural networks for lesion segmentation in brain MRI , 2015 .

[51]  Ronald M. Summers,et al.  DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation , 2015, MICCAI.

[52]  D. Louis Collins,et al.  BEaST: Brain extraction based on nonlocal segmentation technique , 2012, NeuroImage.

[53]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[54]  Lin Yang,et al.  Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation , 2016, NIPS.

[55]  Jieun Kim,et al.  Abdominal multi-organ auto-segmentation using 3D-patch-based deep convolutional neural network , 2020, Scientific Reports.

[56]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Paul A. Yushkevich,et al.  Multi-Atlas Segmentation with Joint Label Fusion , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Hongwei Liu,et al.  Convolutional Neural Network With Data Augmentation for SAR Target Recognition , 2016, IEEE Geoscience and Remote Sensing Letters.

[60]  Shunxing Bao,et al.  SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth , 2018, IEEE Transactions on Medical Imaging.

[61]  Ronald M. Summers,et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms , 2019, ArXiv.