Fingerprints of Generative Models in the Frequency Domain

It is verified in existing works that CNN-based generative models leave unique fingerprints on generated images. There is a lack of analysis about how they are formed in generative models. Interpreting network components in the frequency domain, we derive sources for frequency distribution and grid-like pattern discrepancies exhibited on the spectrum. These insights are leveraged to develop low-cost synthetic models, which generate images emulating the frequency patterns observed in real generative models. The resulting fingerprint extractor pre-trained on synthetic data shows superior transferability in verifying, identifying, and analyzing the relationship of real CNN-based generative models such as GAN, VAE, Flow, and diffusion.

[1]  Juan Cao,et al.  Progressive Open Space Expansion for Open-Set Model Attribution , 2023, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  J. Collomosse,et al.  RepMix: Representation Mixing for Robust Attribution of Synthesized Images , 2022, ECCV.

[3]  Xin Niu,et al.  Learning Convolutional Neural Networks in the Frequency Domain , 2022, ArXiv.

[4]  Prafulla Dhariwal,et al.  Hierarchical Text-Conditional Image Generation with CLIP Latents , 2022, ArXiv.

[5]  Xirong Li,et al.  Deepfake Network Architecture Attribution , 2022, AAAI.

[6]  Yi Ren,et al.  Pseudo Numerical Methods for Diffusion Models on Manifolds , 2022, ICLR.

[7]  B. Ommer,et al.  High-Resolution Image Synthesis with Latent Diffusion Models , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Prafulla Dhariwal,et al.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models , 2021, ICML.

[9]  Fang Wen,et al.  StyleSwin: Transformer-based GAN for High-resolution Image Generation , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Andreas Geiger,et al.  On the Frequency Bias of Generative Models , 2021, NeurIPS.

[11]  Jaakko Lehtinen,et al.  Alias-Free Generative Adversarial Networks , 2021, NeurIPS.

[12]  Younggeun Kim,et al.  Styleformer: Transformer based Generative Adversarial Networks with Style Vector , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Prafulla Dhariwal,et al.  Diffusion Models Beat GANs on Image Synthesis , 2021, NeurIPS.

[14]  Saksham Suri,et al.  Towards Discovery and Attribution of Open-world GAN Generated Images , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  Ngai-Man Cheung,et al.  A Closer Look at Fourier Spectrum Discrepancies for CNN-generated Images Detection , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Drew A. Hudson,et al.  Generative Adversarial Transformers , 2021, ICML.

[17]  Yifan Jiang,et al.  TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up , 2021, NeurIPS.

[18]  Larry S. Davis,et al.  Responsible Disclosure of Generative Models Using Scalable Fingerprinting , 2020, ICLR.

[19]  Changhoon Kim,et al.  Decentralized Attribution of Generative Models , 2020, ICLR.

[20]  Mahyar Khayatkhoei,et al.  Spatial Frequency Bias in Convolutional Generative Adversarial Networks , 2020, AAAI.

[21]  Sahar Abdelnabi,et al.  Artificial GAN Fingerprints: Rooting Deepfake Attribution in Training Data , 2020 .

[22]  Ngai-Man Cheung,et al.  InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[23]  Jaesik Park,et al.  ContraGAN: Contrastive Learning for Conditional Image Generation , 2020, Neural Information Processing Systems.

[24]  Pieter Abbeel,et al.  Denoising Diffusion Probabilistic Models , 2020, NeurIPS.

[25]  Dorothea Kolossa,et al.  Leveraging Frequency Analysis for Deep Fake Image Recognition , 2020, ICML.

[26]  Margret Keuper,et al.  Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Jing Dong,et al.  Scalable Fine-grained Generated Image Classification Based on Deep Metric Learning , 2019, ArXiv.

[28]  Tero Karras,et al.  Analyzing and Improving the Image Quality of StyleGAN , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Tarik Dzanic,et al.  Fourier Spectrum Discrepancies in Deep Network Generated Images , 2019, NeurIPS.

[30]  Ab Al-Hadi Ab Rahman,et al.  Spectral-based convolutional neural network without multiple spatial-frequency domain switchings , 2019, Neurocomputing.

[31]  David Duvenaud,et al.  Residual Flows for Invertible Generative Modeling , 2019, NeurIPS.

[32]  Xiaohua Zhai,et al.  High-Fidelity Image Generation With Fewer Labels , 2019, ICML.

[33]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Mario Fritz,et al.  Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[35]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[36]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[37]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[38]  Arthur Gretton,et al.  Demystifying MMD GANs , 2018, ICLR.

[39]  Shiguang Shan,et al.  AttGAN: Facial Attribute Editing by Only Changing What You Want , 2017, IEEE Transactions on Image Processing.

[40]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[42]  Alireza Fathi,et al.  The Devil is in the Decoder: Classification, Regression and GANs , 2017, International Journal of Computer Vision.

[43]  S. Ermon,et al.  InfoVAE: Information Maximizing Variational Autoencoders , 2017, ArXiv.

[44]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[45]  Vincent Dumoulin,et al.  Deconvolution and Checkerboard Artifacts , 2016 .

[46]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[47]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[48]  Fabrice Labeau,et al.  Discrete Time Signal Processing , 2004 .