Tripartite-GAN: Synthesizing liver contrast-enhanced MRI to improve tumor detection

Contrast-enhanced magnetic resonance imaging (CEMRI) is crucial for the diagnosis of patients with liver tumors, especially for the detection of benign tumors and malignant tumors. However, it suffers from high-risk, time-consuming, and expensive in current clinical diagnosis due to the use of the gadolinium-based contrast agent (CA) injection. If the CEMRI can be synthesized without CA injection, there is no doubt that it will greatly optimize the diagnosis. In this study, we propose a Tripartite Generative Adversarial Network (Tripartite-GAN) as a non-invasive, time-saving, and inexpensive clinical tool by synthesizing CEMRI to detect tumors without CA injection. Specifically, our innovative Tripartite-GAN combines three associated-networks (an attention-aware generator, a convolutional neural network-based discriminator, and a region-based convolutional neural network-based detector) for the first time, which achieves CEMRI synthesis and tumor detection promoting each other in an end-to-end framework. The generator facilitates detector for accurate tumor detection via synthesizing tumor-specific CEMRI. The detector promotes the generator for accurate CEMRI synthesis via the back-propagation. In order to synthesize CEMRI of equivalent clinical value to real CEMRI, the attention-aware generator expands the receptive field via hybrid convolution, and enhances feature representation and context learning of multi-class liver MRI via dual attention mechanism, and improves the performance of convergence of loss via residual learning. Moreover, the attention maps obtained from the generator newly added into the detector improve the performance of tumor detection. The discriminator promotes the generator to synthesize high-quality CEMRI via the adversarial learning strategy. This framework is tested on a large corpus of axial T1 FS Pre-Contrast MRI and axial T1 FS Delay MRI of 265 subjects. Experimental results and quantitative evaluation demonstrate that the Tripartite-GAN achieves high-quality CEMRI synthesis that peak signal-to-noise rate of 28.8 and accurate tumor detection that accuracy of 89.4%, which reveals that Tripartite-GAN can aid in the clinical diagnosis of liver tumors.

[1]  Yogesh Kumar Meena,et al.  Detection of Brain Tumor in MRI Image through Fuzzy-Based Approach , 2018 .

[2]  George Loizou,et al.  Computer vision and pattern recognition , 2007, Int. J. Comput. Math..

[3]  Leonid Karlinsky,et al.  A Region Based Convolutional Network for Tumor Detection and Classification in Breast Mammography , 2016, LABELS/DLMIA@MICCAI.

[4]  Hayit Greenspan,et al.  GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification , 2018, Neurocomputing.

[5]  James M. Brown,et al.  High-resolution medical image synthesis using progressively grown generative adversarial networks , 2018, ArXiv.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[8]  Sang Jun Park,et al.  Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks , 2017, ArXiv.

[9]  Caroline Reinhold,et al.  Liver Tumor Characterization: Comparison Between Liver-specific Gadoxetic Acid Disodium-enhanced MRI and Biphasic CT-A Multicenter Trial , 2006, Journal of computer assisted tomography.

[10]  M. Port,et al.  Clinical and biological consequences of transmetallation induced by contrast agents for magnetic resonance imaging: a review , 2006, Fundamental & clinical pharmacology.

[11]  Ana Maria Mendonça,et al.  End-to-End Adversarial Retinal Image Synthesis , 2018, IEEE Transactions on Medical Imaging.

[12]  D. Sahani,et al.  MRI in detection of hepatocellular carcinoma (HCC) , 2005, Cancer imaging : the official publication of the International Cancer Imaging Society.

[13]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Tom Vercauteren,et al.  Diffeomorphic demons: Efficient non-parametric image registration , 2009, NeuroImage.

[15]  Lone Skov,et al.  Nephrogenic systemic fibrosis: suspected causative role of gadodiamide used for contrast-enhanced magnetic resonance imaging. , 2006, Journal of the American Society of Nephrology : JASN.

[16]  Jost Tobias Springenberg,et al.  Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks , 2015, ICLR.

[17]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[18]  Hayit Greenspan,et al.  Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detection , 2018, Eng. Appl. Artif. Intell..

[19]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[20]  Amritpal Singh,et al.  Detection of brain tumor in MRI images, using combination of fuzzy c-means and SVM , 2015, 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN).

[21]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[22]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  R. Low,et al.  Abdominal MRI advances in the detection of liver tumours and characterisation. , 2007, The Lancet. Oncology.

[24]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Luc Van Gool,et al.  A Three-Player GAN: Generating Hard Samples to Improve Classification Networks , 2019, 2019 16th International Conference on Machine Vision Applications (MVA).

[26]  Garrison W. Cottrell,et al.  Understanding Convolution for Semantic Segmentation , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[27]  Xiangyu Zhang,et al.  Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Ming Dong,et al.  Generating synthetic CTs from magnetic resonance images using generative adversarial networks , 2018, Medical physics.

[29]  Jun Fu,et al.  Dual Attention Network for Scene Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Xiaogang Wang,et al.  FaceID-GAN: Learning a Symmetry Three-Player GAN for Identity-Preserving Face Synthesis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[32]  Qianjin Feng,et al.  Predicting CT Image From MRI Data Through Feature Matching With Learned Nonlinear Local Descriptors , 2018, IEEE Transactions on Medical Imaging.

[33]  Dinggang Shen,et al.  Medical Image Synthesis with Deep Convolutional Adversarial Networks , 2018, IEEE Transactions on Biomedical Engineering.

[34]  Jelmer M. Wolterink,et al.  Deep MR to CT Synthesis Using Unpaired Data , 2017, SASHIMI@MICCAI.

[35]  Jun Zhu,et al.  Triple Generative Adversarial Nets , 2017, NIPS.

[36]  Dimitrios Korkinof,et al.  High-Resolution Mammogram Synthesis using Progressive Generative Adversarial Networks , 2018, ArXiv.

[37]  Michael Lam,et al.  Unsupervised Video Summarization with Adversarial LSTM Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Yichen Wei,et al.  Relation Networks for Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Guang Yang,et al.  Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks , 2017, MIUA.

[40]  Shen Zhao,et al.  Automatic spondylolisthesis grading from MRIs across modalities using faster adversarial recognition network , 2019, Medical Image Anal..