Semi-Supervised Attention-Guided CycleGAN for Data Augmentation on Medical Images

Recently, deep learning methods, in particular, convolutional neural networks (CNNs), have made a massive breakthrough in computer vision. And a big amount of annotated data is the essential cornerstone to reach this success. However, in the medical domain, it is usually difficult (and sometimes even impossible) to get sufficient data for some specific learning tasks. Consequently, in this work, a novel data augmentation solution, called semi-supervised attention-guided CycleGAN (SSA-CycleGAN) is proposed to resolve this problem. Specifically, a cycle-consistency GANs-based model is first proposed to generate synthetic tumor (resp., normal) images from normal (resp., tumor) images. Then, a semi-supervised attention module is further proposed to enhance the model's capability in learning the important details of the training images, which in turns help the generated synthetic images become more realistic. To verify its effectiveness, experimental studies are conducted on three medical image datasets with limited amounts of MRI images, and the proposed SSA-CycleGAN is applied to generate synthetic tumor and normal MRI images for data augmentation. Experimental results show that i) SSA-CycleGAN can add (resp., remove) tumor lesions on (resp., from) the original normal (resp., tumor) images and generate very realistic synthetic tumor (resp. normal) images; and ii) in the ResNet18-based MRI image classification tasks based on these datasets, data augmentation using SSA-CycleGAN achieves much better classification performances than the classic data augmentation methods.

[1]  Shin'ichi Satoh,et al.  Learning More with Less: Conditional PGGAN-based Data Augmentation for Brain Metastases Detection Using Highly-Rough Annotation on MR Images , 2019, CIKM.

[2]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[3]  Hayit Greenspan,et al.  Virtual PET Images from CT Data Using Deep Convolutional Networks: Initial Results , 2017, SASHIMI@MICCAI.

[4]  Lizhuang Ma,et al.  Mask-aware photorealistic facial attribute manipulation , 2018, Computational Visual Media.

[5]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[7]  Dacheng Tao,et al.  Attention-GAN for Object Transfiguration in Wild Images , 2018, ECCV.

[8]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[9]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[10]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[11]  Tanveer F. Syeda-Mahmood,et al.  Chest x-ray generation and data augmentation for cardiovascular abnormality classification , 2018, Medical Imaging.

[12]  Dimitris Kastaniotis,et al.  Attention-Aware Generative Adversarial Networks (ATA-GANs) , 2018, 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[13]  Jeffrey L. Gunter,et al.  Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks , 2018, SASHIMI@MICCAI.

[14]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[15]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[16]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[17]  Eric P. Xing,et al.  Generative Semantic Manipulation with Contrasting GAN , 2017, ArXiv.

[18]  Yuval Elovici,et al.  DOPING: Generative Data Augmentation for Unsupervised Anomaly Detection with GAN , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[19]  Liang Chen,et al.  GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks , 2018, ArXiv.

[20]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.