Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues

As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging research topic of face forgery detection. However, it is extremely challenging since recent advances are able to forge faces beyond the perception ability of human eyes, especially in compressed images and videos. We find that mining forgery patterns with the awareness of frequency could be a cure, as frequency provides a complementary viewpoint where either subtle forgery artifacts or compression errors could be well described. To introduce frequency into the face forgery detection, we propose a novel Frequency in Face Forgery Network (F3-Net), taking advantages of two different but complementary frequency-aware clues, 1) frequency-aware decomposed image components, and 2) local frequency statistics, to deeply mine the forgery patterns via our two-stream collaborative learning framework. We apply DCT as the applied frequency-domain transformation. Through comprehensive studies, we show that the proposed F3-Net significantly outperforms competing state-of-the-art methods on all compression qualities in the challenging FaceForensics++ dataset, especially wins a big lead upon low-quality media.

[1]  Davide Cozzolino,et al.  Image forgery localization through the fusion of camera-based, feature-based and pixel-based techniques , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[2]  Levy Boccato,et al.  Improving image classification with frequency domain layers for feature extraction , 2017, 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP).

[3]  Margret Keuper,et al.  Unmasking DeepFakes with simple Features , 2019, ArXiv.

[4]  Alessandro Piva,et al.  Image Forgery Localization via Fine-Grained Analysis of CFA Artifacts , 2012, IEEE Transactions on Information Forensics and Security.

[5]  Scott McCloskey,et al.  Detecting GAN-Generated Imagery Using Saturation Cues , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[6]  Tieniu Tan,et al.  Live face detection based on the analysis of Fourier spectra , 2004, SPIE Defense + Commercial Sensing.

[7]  Anderson Rocha,et al.  Illuminant-Based Transformed Spaces for Image Forensics , 2016, IEEE Transactions on Information Forensics and Security.

[8]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[9]  Mo Chen,et al.  JPEG-Phase-Aware Convolutional Neural Network for Steganalysis of JPEG Images , 2017, IH&MMSec.

[10]  Hyeonseong Jeon,et al.  FDFtNet: Facing Off Fake Images using Fake Detection Fine-tuning Network , 2020, SEC.

[11]  Davide Cozzolino,et al.  Autoencoder with recurrent neural networks for video forgery detection , 2017, Media Watermarking, Security, and Forensics.

[12]  Justus Thies,et al.  Face2Face: real-time face capture and reenactment of RGB videos , 2019, Commun. ACM.

[13]  Scott McCloskey,et al.  Detecting GAN-generated Imagery using Color Cues , 2018, ArXiv.

[14]  Junichi Yamagishi,et al.  MesoNet: a Compact Facial Video Forgery Detection Network , 2018, 2018 IEEE International Workshop on Information Forensics and Security (WIFS).

[15]  N. Ahmed,et al.  Discrete Cosine Transform , 1996 .

[16]  Alberto Del Bimbo,et al.  Deepfake Video Detection through Optical Flow Based CNN , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[17]  Junichi Yamagishi,et al.  Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Andrew Owens,et al.  CNN-Generated Images Are Surprisingly Easy to Spot… for Now , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Larry S. Davis,et al.  Two-Stream Neural Networks for Tampered Face Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[20]  Xiangyu Zhu,et al.  Deep Spatial Gradient and Temporal Depth Learning for Face Anti-Spoofing , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  D. Sagi,et al.  Gabor filters as texture discriminator , 1989, Biological Cybernetics.

[22]  Justus Thies,et al.  Deferred Neural Rendering: Image Synthesis using Neural Textures , 2019 .

[23]  Fabio Viola,et al.  The Kinetics Human Action Video Dataset , 2017, ArXiv.

[24]  Shaodi You,et al.  A Frequency Domain Neural Network for Fast Image Super-resolution , 2017, 2018 International Joint Conference on Neural Networks (IJCNN).

[25]  Jessica J. Fridrich,et al.  Rich Models for Steganalysis of Digital Images , 2012, IEEE Transactions on Information Forensics and Security.

[26]  Xu Zhang,et al.  Detecting and Simulating Artifacts in GAN Fake Images , 2019, 2019 IEEE International Workshop on Information Forensics and Security (WIFS).

[27]  Chia-Yen Lee,et al.  Learning to Detect Fake Face Images in the Wild , 2018, 2018 International Symposium on Computer, Consumer and Control (IS3C).

[28]  Jitendra Malik,et al.  SlowFast Networks for Video Recognition , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[29]  Tieniu Tan,et al.  Wavelet-SRNet: A Wavelet-Based CNN for Multi-scale Face Super Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Jessica J. Fridrich,et al.  Steganalysis Features for Content-Adaptive JPEG Steganography , 2016, IEEE Transactions on Information Forensics and Security.

[31]  Mario Fritz,et al.  Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  Jianqin Zhou,et al.  On discrete cosine transform , 2011, ArXiv.

[33]  Bin Li,et al.  Detection of Deep Network Generated Images Using Disparities in Color Components , 2018, ArXiv.

[34]  Baining Guo,et al.  Face X-Ray for More General Face Forgery Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Davide Cozzolino,et al.  Detection of GAN-Generated Fake Images over Social Networks , 2018, 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[36]  Kaushal K. Shukla,et al.  Passive forensics in image and video using noise features: A review , 2016, Digit. Investig..

[37]  M. Bodruzzaman,et al.  Feature extraction using wavelet transform for neural network based image classification , 1998, Proceedings of Thirtieth Southeastern Symposium on System Theory.

[38]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[41]  P. M. Bentley,et al.  Wavelet transforms: an introduction , 1994 .

[42]  B. S. Manjunath,et al.  Rotation-invariant texture classification using a complete space-frequency model , 1999, IEEE Trans. Image Process..

[43]  Chia-Wen Lin,et al.  Video forgery detection using correlation of noise residue , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[44]  Davide Cozzolino,et al.  Recasting Residual-based Local Descriptors as Convolutional Neural Networks: an Application to Image Forgery Detection , 2017, IH&MMSec.

[45]  Shaziya .P.S. Khan,et al.  Exposing Digital Image Forgeries by Illumination Color Classification , 2015 .

[46]  Junichi Yamagishi,et al.  Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos , 2019, 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[47]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[48]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[49]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[50]  Andreas Rössler,et al.  FaceForensics++: Learning to Detect Manipulated Facial Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[51]  Bin Li,et al.  Identification of deep network generated images using disparities in color components , 2020, Signal Process..

[52]  Xing Zhang,et al.  Exposing image splicing with inconsistent local noise variances , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[53]  Premkumar Natarajan,et al.  Recurrent Convolutional Strategies for Face Manipulation Detection in Videos , 2019, CVPR Workshops.

[54]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Anis Nurashikin Nordin,et al.  Development of photo forensics algorithm by detecting photoshop manipulation using error level analysis , 2017 .

[56]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[57]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[58]  Toshiya Hachisuka,et al.  Wavelet Convolutional Neural Networks for Texture Classification , 2017, ArXiv.

[59]  Florian Franzen.  Image Classification in the Frequency Domain with Neural Networks and Absolute Value DCT , 2018, ICISP.

[60]  Junichi Yamagishi,et al.  Distinguishing computer graphics from natural images using convolution neural networks , 2017, 2017 IEEE Workshop on Information Forensics and Security (WIFS).

[61]  Junichi Yamagishi,et al.  Use of a Capsule Network to Detect Fake Images and Videos , 2019, ArXiv.

[62]  Ying Huang,et al.  Deep Frequent Spatial Temporal Learning for Face Anti-Spoofing , 2020, ArXiv.

[63]  Belhassen Bayar,et al.  A Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer , 2016, IH&MMSec.