Boosting segmentation with weak supervision from image-to-image translation

In many cases, especially with medical images, it is prohibitively challenging to produce a sufficiently large training sample of pixel-level annotations to train deep neural networks for semantic image segmentation. On the other hand, some information is often known about the contents of images. We leverage information on whether an image presents the segmentation target or whether it is absent from the image to improve segmentation performance by augmenting the amount of data usable for model training. Specifically, we propose a semi-supervised framework that employs image-to-image translation between weak labels (e.g., presence vs. absence of cancer), in addition to fully supervised segmentation on some examples. We conjecture that this translation objective is well aligned with the segmentation objective as both require the same disentangling of image variations. Building on prior image-to-image translation work, we re-use the encoder and decoders for translating in either direction between two domains, employing a strategy of selectively decoding domain-specific variations. For presence vs. absence domains, the encoder produces variations that are common to both and those unique to the presence domain. Furthermore, we successfully re-use one of the decoders used in translation for segmentation. We validate the proposed method on synthetic tasks of varying difficulty as well as on the real task of brain tumor segmentation in magnetic resonance images, where we show significant improvements over standard semi-supervised training with autoencoding.

[1]  Klaus H. Maier-Hein,et al.  Adversarial Networks for the Detection of Aggressive Prostate Cancer , 2017, ArXiv.

[2]  Ender Konukoglu,et al.  Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders , 2018, ArXiv.

[3]  Jinhua Yu,et al.  Brain Tumor Segmentation Using an Adversarial Network , 2017, BrainLes@MICCAI.

[4]  Andriy Myronenko,et al.  3D MRI brain tumor segmentation using autoencoder regularization , 2018, BrainLes@MICCAI.

[5]  Hayit Greenspan,et al.  GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification , 2018, Neurocomputing.

[6]  Victor S. Lempitsky,et al.  Unsupervised Domain Adaptation by Backpropagation , 2014, ICML.

[7]  Nassir Navab,et al.  Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images , 2018, BrainLes@MICCAI.

[8]  Tao Xu,et al.  SegAN: Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation , 2017, Neuroinformatics.

[9]  Albert C. S. Chung,et al.  Learning Data Augmentation for Brain Tumor Segmentation with Coarse-to-Fine Generative Adversarial Networks , 2018, BrainLes@MICCAI.

[10]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[12]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[13]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[14]  Youbao Tang,et al.  CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation , 2018, MICCAI.

[15]  Christoph Meinel,et al.  Whole Heart and Great Vessel Segmentation with Context-aware of Generative Adversarial Networks , 2018, Bildverarbeitung für die Medizin.

[16]  Xiaohui Xie,et al.  Adversarial Deep Structural Networks for Mammographic Mass Segmentation , 2016, bioRxiv.

[17]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[18]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[19]  Philip Bachman,et al.  Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data , 2018, ICML.

[20]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[21]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Daguang Xu,et al.  Automatic Liver Segmentation Using an Adversarial Image-to-Image Network , 2017, MICCAI.

[23]  Simon Andermatt,et al.  Pathology Segmentation using Distributional Differences to Images of Healthy Origin , 2018, BrainLes@MICCAI.

[24]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[25]  John T. Guibas,et al.  Synthetic Medical Images from Dual Generative Adversarial Networks , 2017, ArXiv.

[26]  Ender Konukoglu,et al.  Visual Feature Attribution Using Wasserstein GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Maneesh Kumar Singh,et al.  DRIT++: Diverse Image-to-Image Translation via Disentangled Representations , 2019, International Journal of Computer Vision.

[29]  Yuan Li,et al.  SCAN: Structure Correcting Adversarial Network for Chest X-rays Organ Segmentation , 2017, ArXiv.

[30]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[31]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[32]  Ghassan Hamarneh,et al.  Generative adversarial networks to segment skin lesions , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[33]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[35]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[36]  Georg Langs,et al.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery , 2017, IPMI.

[37]  Sang Jun Park,et al.  Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks , 2017, ArXiv.

[38]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[39]  Sanjiv Kumar,et al.  On the Convergence of Adam and Beyond , 2018 .

[40]  Mitko Veta,et al.  Adversarial Training and Dilated Convolutions for Brain MRI Segmentation , 2017, DLMIA/ML-CDS@MICCAI.

[41]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[42]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[43]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[44]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[45]  Bernhard Schölkopf,et al.  On causal and anticausal learning , 2012, ICML.

[46]  Lin Yang,et al.  Deep Adversarial Networks for Biomedical Image Segmentation Utilizing Unannotated Images , 2017, MICCAI.