Free-form tumor synthesis in computed tomography images via richer generative adversarial network

The insufficiency of annotated medical imaging scans for cancer makes it challenging to train and validate data-hungry deep learning models in precision oncology. We propose a new richer generative adversarial network for free-form 3D tumor/lesion synthesis in computed tomography (CT) images. The network is composed of a new richer convolutional feature enhanced dilated-gated generator (RicherDG) and a hybrid loss function. The RicherDG has dilated-gated convolution layers to enable tumor-painting and to enlarge perceptive fields; and it has a novel richer convolutional feature association branch to recover multi-scale convolutional features especially from uncertain boundaries between tumor and surrounding healthy tissues. The hybrid loss function, which consists of a diverse range of losses, is designed to aggregate complementary information to improve optimization. We perform a comprehensive evaluation of the synthesis results on a wide range of public CT image datasets covering the liver, kidney tumors, and lung nodules. The qualitative and quantitative evaluations and ablation study demonstrated improved synthesizing results over advanced tumor synthesis methods.

[1]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Junqing Yu,et al.  Adversarial Style Mining for One-Shot Unsupervised Domain Adaptation , 2020, NeurIPS.

[3]  Youbao Tang,et al.  CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation , 2018, MICCAI.

[4]  Ralph V Clayman,et al.  Laparoscopic partial nephrectomy for renal masses: effect of tumor location. , 2006, Urology.

[5]  Allan Hanbury,et al.  An Efficient Algorithm for Calculating the Exact Hausdorff Distance , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Xiang Bai,et al.  Richer Convolutional Features for Edge Detection , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Huiqi Li,et al.  Synthesizing retinal and neuronal images with generative adversarial nets , 2018, Medical Image Anal..

[8]  Mehran Ebrahimi,et al.  EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning , 2019, ArXiv.

[9]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[10]  Hayit Greenspan,et al.  Synthetic data augmentation using GAN for improved liver lesion classification , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[11]  Zichen Zhang,et al.  U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection , 2020, Pattern Recognit..

[12]  Giancarlo Mauri,et al.  GAN-based synthetic brain MR image generation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[13]  Nikolaos Papanikolopoulos,et al.  The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes , 2019, ArXiv.

[14]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Hao Chen,et al.  The Liver Tumor Segmentation Benchmark (LiTS) , 2019, Medical Image Anal..

[16]  Aboul Ella Hassanien,et al.  3D brain tumor segmentation scheme using K-mean clustering and connected component labeling algorithms , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[17]  Yi Yang,et al.  Macro-Micro Adversarial Network for Human Parsing , 2018, ECCV.

[18]  Jaime S. Cardoso,et al.  Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support , 2017, Lecture Notes in Computer Science.

[19]  Giancarlo Mauri,et al.  Combining Noise-to-Image and Image-to-Image GANs: Brain MR Image Augmentation for Tumor Detection , 2019, IEEE Access.

[20]  Sang Won Yoon,et al.  Medical Image Synthesis with Generative Adversarial Networks for Tissue Recognition , 2018, 2018 IEEE International Conference on Healthcare Informatics (ICHI).

[21]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[22]  G. Fitzgerald,et al.  'I. , 2019, Australian journal of primary health.

[23]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[24]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[25]  David Cox,et al.  Conditional Infilling GANs for Data Augmentation in Mammogram Classification , 2018, RAMBO+BIA+TIA@MICCAI.

[26]  Kai Ma,et al.  Med3D: Transfer Learning for 3D Medical Image Analysis , 2019, ArXiv.

[27]  Hao Chen,et al.  Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge , 2016, Medical Image Anal..

[28]  C. Mathers,et al.  Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008 , 2010, International journal of cancer.

[29]  Thomas Neff,et al.  Generative Adversarial Network based Synthesis for Supervised Medical Image Segmentation , 2017 .

[30]  Hayit Greenspan,et al.  Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detection , 2018, Eng. Appl. Artif. Intell..

[31]  Winston H. Hsu,et al.  Free-Form Video Inpainting With 3D Gated Convolution and Temporal PatchGAN , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  Fei Yang,et al.  Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Frank Zijlstra,et al.  CT synthesis from MR images for orthopedic applications in the lower arm using a conditional generative adversarial network , 2019, Medical Imaging: Image Processing.

[34]  Ghassan Hamarneh,et al.  Mask2Lesion: Mask-Constrained Adversarial Skin Lesion Image Synthesis , 2019, SASHIMI@MICCAI.

[35]  Ting-Chun Wang,et al.  Image Inpainting for Irregular Holes Using Partial Convolutions , 2018, ECCV.

[36]  Kristen Grauman,et al.  Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Ulas Bagci,et al.  How to fool radiologists with generative adversarial networks? A visual turing test for lung cancer diagnosis , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[38]  Thomas de Quincey.  [C] , 2000, The Works of Thomas De Quincey, Vol. 1: Writings, 1799–1820.

[39]  Jelmer M. Wolterink,et al.  Deep MR to CT Synthesis Using Unpaired Data , 2017, SASHIMI@MICCAI.

[40]  Kunio Doi,et al.  Computer-aided diagnosis in medical imaging: Historical review, current status and future potential , 2007, Comput. Medical Imaging Graph..

[41]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[42]  Raymond Y Huang,et al.  Artificial intelligence in cancer imaging: Clinical challenges and applications , 2019, CA: a cancer journal for clinicians.

[43]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[44]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[45]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.