D3T-GAN: Data-Dependent Domain Transfer GANs for Image Generation with Limited Data

As an important and challenging problem, image generation with limited data aims at generating realistic images through training a GAN model given few samples. A typical solution is to transfer a well-trained GAN model from a data-rich source domain to the data-deficient target domain. In this paper, we propose a novel self-supervised transfer scheme termed D3T-GAN, addressing the cross-domain GANs transfer in limited image generation. Specifically, we design two individual strategies to transfer knowledge between generators and discriminators, respectively. To transfer knowledge between generators, we conduct a data-dependent transformation, which projects target samples into the latent space of source generator and reconstructs them back. Then, we perform knowledge transfer from transformed samples to generated samples. To transfer knowledge between discriminators, we design a multi-level discriminant knowledge distillation from the source discriminator to the target discriminator on both the real and fake samples. Extensive experiments show that our method improves the quality of generated images and achieves the state-of-the-art FID scores on commonly used datasets.

[1]  Bolei Zhou,et al.  Data-Efficient Instance Generation from Instance Discrimination , 2021, NeurIPS.

[2]  Aniwat Phaphuangwittayakul,et al.  Fast Adaptive Meta-Learning for Few-Shot Image Generation , 2021, IEEE Transactions on Multimedia.

[3]  Linchao Zhu,et al.  Few-Shot Common-Object Reasoning Using Common-Centric Localization Network , 2021, IEEE Transactions on Image Processing.

[4]  Chenyou Fan,et al.  Federated Few-Shot Learning with Adversarial Learning , 2021, 2021 19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt).

[5]  Balaji Krishnamurthy,et al.  Data Instance Prior for Transfer Learning in GANs , 2020, ArXiv.

[6]  Eli Shechtman,et al.  Few-shot Image Generation with Elastic Weight Consolidation , 2020, NeurIPS.

[7]  Weili Guan,et al.  A Pairwise Attentive Adversarial Spatiotemporal Network for Cross-Domain Few-Shot Action Recognition-R2 , 2020, IEEE Transactions on Image Processing.

[8]  Zhaorui Gu,et al.  Few-shot Fish Image Generation and Classification , 2020, Global Oceans 2020: Singapore – U.S. Gulf Coast.

[9]  Abhishek Kumar,et al.  Few-Shot Adaptation of Generative Adversarial Networks , 2020, ArXiv.

[10]  Yan Hong,et al.  F2GAN: Fusing-and-Filling GAN for Few-shot Image Generation , 2020, ACM Multimedia.

[11]  Shuqiang Jiang,et al.  Few-shot Food Recognition via Multi-view Representation Learning , 2020, ACM Trans. Multim. Comput. Commun. Appl..

[12]  John Jethro Virtusio,et al.  Sketch-guided Deep Portrait Generation , 2020, ACM Trans. Multim. Comput. Commun. Appl..

[13]  Song Han,et al.  Differentiable Augmentation for Data-Efficient GAN Training , 2020, NeurIPS.

[14]  Ngai-Man Cheung,et al.  On Data Augmentation for GAN Training , 2020, IEEE Transactions on Image Processing.

[15]  Sameer Singh,et al.  Image Augmentations for GAN Training , 2020, ArXiv.

[16]  Tero Karras,et al.  Training Generative Adversarial Networks with Limited Data , 2020, NeurIPS.

[17]  Bolei Zhou,et al.  In-Domain GAN Inversion for Real Image Editing , 2020, ECCV.

[18]  Yan Hong,et al.  Matchinggan: Matching-Based Few-Shot Image Generation , 2020, 2020 IEEE International Conference on Multimedia and Expo (ICME).

[19]  L. Carin,et al.  On Leveraging Pretrained GANs for Generation with Limited Data , 2020, ICML.

[20]  Jinwoo Shin,et al.  Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs , 2020, 2002.10964.

[21]  Honglak Lee,et al.  Improved Consistency Regularization for GANs , 2020, AAAI.

[22]  WangYunhong,et al.  U-Net Conditional GANs for Photo-Realistic and Identity-Preserving Facial Expression Synthesis , 2020 .

[23]  Zixuan Liu,et al.  DAWSON: A Domain Adaptive Few Shot Generation Framework , 2020, ArXiv.

[24]  Bolei Zhou,et al.  Image Processing Using Multi-Code GAN Prior , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Nicu Sebe,et al.  Unified Generative Adversarial Networks for Controllable Image-to-Image Translation , 2019, IEEE Transactions on Image Processing.

[26]  Joost van de Weijer,et al.  MineGAN: Effective Knowledge Transfer From GANs to Target Domains With Few Images , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Jan Kautz,et al.  Few-shot Video-to-Video Synthesis , 2019, NeurIPS.

[29]  Honglak Lee,et al.  Consistency Regularization for Generative Adversarial Networks , 2019, ICLR.

[30]  Yunhong Wang,et al.  U-Net Conditional GANs for Photo-Realistic and Identity-Preserving Facial Expression Synthesis , 2019 .

[31]  Sheng Tang,et al.  Asymmetric GAN for Unpaired Image-to-Image Translation , 2019, IEEE Transactions on Image Processing.

[32]  Quanming Yao,et al.  Few-shot Learning: A Survey , 2019, ArXiv.

[33]  Peter Wonka,et al.  Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Tatsuya Harada,et al.  Image Generation From Small Datasets via Batch Statistics Adaptation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[35]  Louis Clouâtre,et al.  FIGR: Few-shot Image Generation with Reptile , 2019, ArXiv.

[36]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[38]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[39]  Bogdan Raducanu,et al.  Transferring GANs: generating images from limited data , 2018, ECCV.

[40]  Dmitry P. Vetrov,et al.  Few-shot Generative Modelling with Generative Matching Networks , 2018, AISTATS.

[41]  Joshua Achiam,et al.  On First-Order Meta-Learning Algorithms , 2018, ArXiv.

[42]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[43]  Sebastian Nowozin,et al.  Which Training Methods for GANs do actually Converge? , 2018, ICML.

[44]  Amos J. Storkey,et al.  Data Augmentation Generative Adversarial Networks , 2017, ICLR 2018.

[45]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[46]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[47]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[48]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[49]  Alexei A. Efros,et al.  Generative Visual Manipulation on the Natural Image Manifold , 2016, ECCV.

[50]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[51]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[52]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[53]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[55]  Yinda Zhang,et al.  LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop , 2015, ArXiv.

[56]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[57]  Song-Chun Zhu,et al.  Learning Hybrid Image Templates (HIT) by Information Projection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.