Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training

Conditional Generative Adversarial Networks (cGAN) generate realistic images by incorporating class information into GAN. While one of the most popular cGANs is an auxiliary classifier GAN with softmax cross-entropy loss (ACGAN), it is widely known that training ACGAN is challenging as the number of classes in the dataset increases. ACGAN also tends to generate easily classifiable samples with a lack of diversity. In this paper, we introduce two cures for ACGAN. First, we identify that gradient exploding in the classifier can cause an undesirable collapse in early training, and projecting input vectors onto a unit hypersphere can resolve the problem. Second, we propose the Data-to-Data Cross-Entropy loss (D2D-CE) to exploit relational information in the class-labeled dataset. On this foundation, we propose the Rebooted Auxiliary Classifier Generative Adversarial Network (ReACGAN). The experimental results show that ReACGAN achieves state-of-the-art generation results on CIFAR10, Tiny-ImageNet, CUB200, and ImageNet datasets. We also verify that ReACGAN benefits from differentiable augmentations and that D2D-CE harmonizes with StyleGAN2 architecture. Model weights and a software package that provides implementations of representative cGANs and all experiments in our paper are available at https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.

[1]  Matthew R. Walter,et al.  Boosting Contrastive Self-Supervised Learning with False Negative Cancellation , 2020, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[2]  Saeid Nahavandi,et al.  Deep learning for deepfakes creation and detection: A survey , 2019, Comput. Vis. Image Underst..

[3]  Asim Kadav,et al.  Dual Projection Generative Adversarial Networks for Conditional Image Generation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  Jinwoo Shin,et al.  Training GANs with Stronger Augmentations via Contrastive Discriminator , 2021, ICLR.

[5]  Prafulla Dhariwal,et al.  Improved Denoising Diffusion Probabilistic Models , 2021, ICML.

[6]  Wojciech Czaja,et al.  A Multi-Class Hinge Loss for Conditional GANs , 2021, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[7]  Bingbing Ni,et al.  Omni-GAN: On the Secrets of cGANs and Beyond , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  S. Sra,et al.  Contrastive Learning with Hard Negative Samples , 2020, ICLR.

[9]  Bolei Zhou,et al.  Closed-Form Factorization of Latent Semantics in GANs , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Honglak Lee,et al.  Improved Consistency Regularization for GANs , 2020, AAAI.

[11]  Xueqi Cheng,et al.  cGANs with Auxiliary Discriminative Classifier , 2021, ArXiv.

[12]  Artem Babenko,et al.  On Self-Supervised Image Representations for GAN Evaluation , 2021, ICLR.

[13]  Il-Chul Moon,et al.  Score Matching Model for Unbounded Data Score , 2021, ArXiv.

[14]  Minsu Cho,et al.  CircleGAN: Generative Adversarial Learning across Spherical Circles , 2020, NeurIPS.

[15]  Iacopo Masi,et al.  Two-branch Recurrent Network for Isolating Deepfakes in Videos , 2020, ECCV.

[16]  Ching-Yao Chuang,et al.  Debiased Contrastive Learning , 2020, NeurIPS.

[17]  Jaesik Park,et al.  ContraGAN: Contrastive Learning for Conditional Image Generation , 2020, Neural Information Processing Systems.

[18]  Pierre H. Richemond,et al.  Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , 2020, NeurIPS.

[19]  Tero Karras,et al.  Training Generative Adversarial Networks with Limited Data , 2020, NeurIPS.

[20]  Fahad Shahbaz Khan,et al.  A Self-supervised Approach for Adversarial Robustness , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Yoshua Bengio,et al.  Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling , 2020, NeurIPS.

[22]  Jinwoo Shin,et al.  Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs , 2020, 2002.10964.

[23]  Seong Joon Oh,et al.  Reliable Fidelity and Diversity Metrics for Generative Models , 2020, ICML.

[24]  Colin Raffel,et al.  Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples , 2020, NeurIPS.

[25]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[26]  Jung-Woo Ha,et al.  StarGAN v2: Diverse Image Synthesis for Multiple Domains , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Honglak Lee,et al.  Consistency Regularization for Generative Adversarial Networks , 2019, ICLR.

[29]  Yan Wu,et al.  LOGAN: Latent Optimisation for Generative Adversarial Networks , 2019, ArXiv.

[30]  Kun Zhang,et al.  Twin Auxilary Classifiers GAN , 2019, NeurIPS.

[31]  Suman V. Ravuri,et al.  Classification Accuracy Score for Conditional Generative Models , 2019, NeurIPS.

[32]  Jaakko Lehtinen,et al.  Few-Shot Unsupervised Image-to-Image Translation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[33]  Ning Xu,et al.  Controllable Artistic Text Style Transfer via Shape-Matching GAN , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Jaakko Lehtinen,et al.  Improved Precision and Recall Metric for Assessing Generative Models , 2019, NeurIPS.

[35]  Lantao Yu,et al.  Lipschitz Generative Adversarial Nets , 2019, ICML.

[36]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[38]  Stefan Winkler,et al.  The Unusual Effectiveness of Averaging in GAN Training , 2018, ICLR.

[39]  Nicu Sebe,et al.  Whitening and Coloring Batch Transform for GANs , 2018, ICLR.

[40]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[41]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Bernt Schiele,et al.  Adversarial Scene Editing: Automatic Object Removal from Weak Supervision , 2018, NeurIPS.

[43]  Olivier Bachem,et al.  Assessing Generative Models via Precision and Recall , 2018, NeurIPS.

[44]  Lu Zhang,et al.  FairGAN: Fairness-aware Generative Adversarial Networks , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[45]  Takeru Miyato,et al.  cGANs with Projection Discriminator , 2018, ICLR.

[46]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[47]  Jacob Abernethy,et al.  On Convergence and Stability of GANs , 2018 .

[48]  Yong-Jin Liu,et al.  CartoonGAN: Generative Adversarial Networks for Photo Cartoonization , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  James Hays,et al.  SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[50]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[51]  Hao Wu,et al.  Mixed Precision Training , 2017, ICLR.

[52]  Yong Yu,et al.  Activation Maximization Generative Adversarial Nets , 2017 .

[53]  Hugo Larochelle,et al.  Modulating early visual processing by language , 2017, NIPS.

[54]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[55]  Charles A. Sutton,et al.  VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning , 2017, NIPS.

[56]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[57]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[58]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[59]  Jonathon Shlens,et al.  Conditional Image Synthesis with Auxiliary Classifier GANs , 2016, ICML.

[60]  Jonathon Shlens,et al.  A Learned Representation For Artistic Style , 2016, ICLR.

[61]  Minh N. Do,et al.  Semantic Image Inpainting with Deep Generative Models , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[63]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[64]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[66]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[67]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[68]  Ya Le,et al.  Tiny ImageNet Visual Recognition Challenge , 2015 .

[69]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[70]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[71]  Pietro Perona,et al.  Caltech-UCSD Birds 200 , 2010 .

[72]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[73]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .