Unsupervised Shadow Removal Using Target Consistency Generative Adversarial Network

Unsupervised shadow removal aims to learn a non-linear function to map the original image from shadow domain to non-shadow domain in the absence of paired shadow and non-shadow data. In this paper, we develop a simple yet efficient target-consistency generative adversarial network (TC-GAN) for the shadow removal task in the unsupervised manner. Compared with the bidirectional mapping in cycle-consistency GAN based methods for shadow removal, TC-GAN tries to learn a one-sided mapping to cast shadow images into shadow-free ones. With the proposed target-consistency constraint, the correlations between shadow images and the output shadow-free image are strictly confined. Extensive comparison experiments results show that TC-GAN outperforms the state-of-the-art unsupervised shadow removal methods by 14.9% in terms of FID and 31.5% in terms of KID. It is rather remarkable that TC-GAN achieves comparable performance with supervised shadow removal methods.

[1]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[2]  Harry Shum,et al.  Natural shadow matting , 2007, TOGS.

[3]  D. Tao,et al.  Deep Domain Generalization via Conditional Invariant Adversarial Networks , 2018, ECCV.

[4]  Nicu Sebe,et al.  Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Han Gong,et al.  Interactive Shadow Removal and Ground Truth for Variable Scene Categories , 2014, BMVC.

[7]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[9]  Rynson W. H. Lau,et al.  DeshadowNet: A Multi-context Embedding Deep Network for Shadow Removal , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Neil A. Dodgson,et al.  Proceedings Ninth IEEE International Conference on Computer Vision , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[11]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[12]  Chi-Wing Fu,et al.  Mask-ShadowGAN: Learning to Remove Shadows From Unpaired Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Ping Tan,et al.  DualGAN: Unsupervised Dual Learning for Image-to-Image Translation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Alexei A. Efros,et al.  Toward Multimodal Image-to-Image Translation , 2017, NIPS.

[15]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Arthur Gretton,et al.  Demystifying MMD GANs , 2018, ICLR.

[17]  Dimitris Samaras,et al.  Leave-One-Out Kernel Optimization for Shadow Detection and Removal , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[19]  Chi-Wing Fu,et al.  Direction-Aware Spatial Context Features for Shadow Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Luc Van Gool,et al.  ComboGAN: Unrestrained Scalability for Image Domain Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Kwang In Kim,et al.  Unsupervised Attention-guided Image to Image Translation , 2018, NeurIPS.

[22]  Kun Zhang,et al.  Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  Luc Van Gool,et al.  European conference on computer vision (ECCV) , 2006, eccv 2006.

[25]  Michael Gleicher,et al.  Texture-Consistent Shadow Removal , 2008, ECCV.

[26]  Derek Hoiem,et al.  Paired Regions for Shadow Detection and Removal , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Michael Terry,et al.  Learning to Remove Soft Shadows , 2015, ACM Trans. Graph..

[28]  Nicu Sebe,et al.  AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networks , 2019, ArXiv.

[29]  Chunxia Xiao,et al.  Shadow Inpainting and Removal Using Generative Adversarial Networks with Slice Convolutions , 2019, Comput. Graph. Forum.

[30]  Kwan-Liu Ma,et al.  Fast Shadow Removal Using Adaptive Multi‐Scale Illumination Transfer , 2013, Comput. Graph. Forum.

[31]  Cheng Lu,et al.  On the removal of shadows from images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Hyunsoo Kim,et al.  Learning to Discover Cross-Domain Relations with Generative Adversarial Networks , 2017, ICML.

[33]  Wenbin Cai,et al.  Separating Style and Content for Generalized Style Transfer , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Minjae Kim,et al.  U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation , 2019, ICLR.

[35]  Chunxia Xiao,et al.  ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[36]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[37]  Chi-Wing Fu,et al.  Bidirectional Feature Pyramid Network with Recurrent Attention Residual Modules for Shadow Detection , 2018, ECCV.

[38]  Le Hui,et al.  Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Lior Wolf,et al.  Unsupervised Cross-Domain Image Generation , 2016, ICLR.

[40]  Huchuan Lu,et al.  Deep Mutual Learning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Chunxia Xiao,et al.  RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal , 2020, AAAI.

[42]  Wei Liu,et al.  Unsupervised Image-to-Image Translation with Stacked Cycle-Consistent Adversarial Networks , 2018, ECCV.

[43]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[44]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Chao Yang,et al.  Show, Attend, and Translate: Unsupervised Image Translation With Self-Regularization and Attention , 2018, IEEE Transactions on Image Processing.

[46]  Narendra Ahuja,et al.  Shadow Removal Using Bilateral Filtering , 2012, IEEE Transactions on Image Processing.

[47]  Bernt Schiele,et al.  Generative Adversarial Text to Image Synthesis , 2016, ICML.

[48]  Jack Tumblin,et al.  Editing Soft Shadows in a Digital Photograph , 2007, IEEE Computer Graphics and Applications.

[49]  Oleksii Sidorov,et al.  Conditional GANs for Multi-Illuminant Color Constancy: Revolution or yet Another Approach? , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[50]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Mohammed Bennamoun,et al.  Automatic Shadow Detection and Removal from a Single Image , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  H KhanSalman,et al.  Automatic Shadow Detection and Removal from a Single Image , 2016 .

[53]  Cheng Shi,et al.  Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN , 2019, AAAI.