Deep Learning for Deepfakes Creation and Detection

Deep learning has been successfully applied to solve various complex problems ranging from big data analytics to computer vision and human-level control. Deep learning advances however have also been employed to create software that can cause threats to privacy, democracy and national security. One of those deep learning-powered applications recently emerged is “deepfake”. Deepfake algorithms can create fake images and videos that humans cannot distinguish them from authentic ones. The proposal of technologies that can automatically detect and assess the integrity of digital visual media is therefore indispensable. This paper presents a survey of algorithms used to create deepfakes and, more importantly, methods proposed to detect deepfakes in the literature to date. We present extensive discussions on challenges, research trends and directions related to deepfake technologies. By reviewing the background of deepfakes and state-of-the-art deepfake detection methods, this study provides a comprehensive overview of deepfake techniques and facilitates the development of new and more robust methods to deal with the increasingly challenging deepfakes.

[1]  Jonathan G. Fiscus,et al.  MFC Datasets: Large-Scale Benchmark Datasets for Media Forensic Challenge Evaluation , 2019, 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[2]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Junichi Yamagishi,et al.  Distinguishing computer graphics from natural images using convolution neural networks , 2017, 2017 IEEE Workshop on Information Forensics and Security (WIFS).

[4]  Miroslav Goljan,et al.  Digital camera identification from sensor pattern noise , 2006, IEEE Transactions on Information Forensics and Security.

[5]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[6]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[7]  Belhassen Bayar,et al.  A Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer , 2016, IH&MMSec.

[8]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[9]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[10]  Andreas Rössler,et al.  FaceForensics++: Learning to Detect Manipulated Facial Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Sébastien Marcel,et al.  Speaker Inconsistency Detection in Tampered Video , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[12]  Pablo Garrido,et al.  High-Fidelity Monocular Face Reconstruction Based on an Unsupervised Model-Based Face Autoencoder , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Sébastien Marcel,et al.  Vulnerability assessment and detection of Deepfake videos , 2019, 2019 International Conference on Biometrics (ICB).

[14]  Davide Cozzolino,et al.  Detection of GAN-Generated Fake Images over Social Networks , 2018, 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[15]  Anil K. Jain,et al.  Face Spoof Detection With Image Distortion Analysis , 2015, IEEE Transactions on Information Forensics and Security.

[16]  Edward J. Delp,et al.  Deepfake Video Detection Using Recurrent Neural Networks , 2018, 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[17]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[18]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[19]  Marco Fontani,et al.  A Video Forensic Framework for the Unsupervised Analysis of MP4-Like File Container , 2019, IEEE Transactions on Information Forensics and Security.

[20]  Chih-Chung Hsu,et al.  Deep Fake Image Detection Based on Pairwise Learning , 2020, Applied Sciences.

[21]  Sakshi Agarwal,et al.  Limits of Deepfake Detection: A Robust Estimation Viewpoint , 2019, ArXiv.

[22]  Siwei Lyu,et al.  In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking , 2018, 2018 IEEE International Workshop on Information Forensics and Security (WIFS).

[23]  Joon Son Chung,et al.  Lip Reading Sentences in the Wild , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[25]  Christian Riess,et al.  Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations , 2019, 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[26]  Junichi Yamagishi,et al.  MesoNet: a Compact Facial Video Forgery Detection Network , 2018, 2018 IEEE International Workshop on Information Forensics and Security (WIFS).

[27]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[28]  Marie-Helen Maras,et al.  Determining authenticity of video evidence in the age of artificial intelligence and in the wake of Deepfake videos , 2018, The International Journal of Evidence & Proof.

[29]  Jessica J. Fridrich,et al.  Rich Models for Steganalysis of Digital Images , 2012, IEEE Transactions on Information Forensics and Security.

[30]  Jing Dong,et al.  On the generalization of GAN image forensics , 2019, CCBR.

[31]  H. Farid,et al.  Image forgery detection , 2009, IEEE Signal Processing Magazine.

[32]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[33]  Jianmei Yang,et al.  A Fast Forgery Detection Algorithm Based on Exponential-Fourier Moments for Video Region Duplication , 2018, IEEE Transactions on Multimedia.

[34]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[35]  Xin Yang,et al.  Exposing Deep Fakes Using Inconsistent Head Poses , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[36]  Sébastien Marcel,et al.  On the effectiveness of local binary patterns in face anti-spoofing , 2012, 2012 BIOSIG - Proceedings of the International Conference of Biometrics Special Interest Group (BIOSIG).

[37]  Ying Zhang,et al.  Automated face swapping and its detection , 2017, 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP).

[38]  Sébastien Marcel,et al.  Face Anti-spoofing Based on General Image Quality Assessment , 2014, 2014 22nd International Conference on Pattern Recognition.

[39]  Zhenan Sun,et al.  3D Aided Duet GANs for Multi-View Face Image Synthesis , 2019, IEEE Transactions on Information Forensics and Security.

[40]  Bolin Chen,et al.  Fake Faces Identification via Convolutional Neural Network , 2018, IH&MMSec.

[41]  Siwei Lyu,et al.  Exposing DeepFake Videos By Detecting Face Warping Artifacts , 2018, CVPR Workshops.

[42]  Ron J. Weiss,et al.  Unsupervised Speech Representation Learning Using WaveNet Autoencoders , 2019, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[43]  Fang Liu,et al.  Task-Oriented GAN for PolSAR Image Classification and Clustering , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[44]  Jing Dong,et al.  Deep learning for steganalysis via convolutional neural networks , 2015, Electronic Imaging.

[45]  Qingmin Liao,et al.  FV-GAN: Finger Vein Representation Using Generative Adversarial Networks , 2019, IEEE Transactions on Information Forensics and Security.

[46]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[47]  Junichi Yamagishi,et al.  Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[48]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[49]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[50]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[51]  Husrev T. Sencar,et al.  A study of the robustness of PRNU-based camera identification , 2009, Electronic Imaging.

[52]  Abhijith Punnappurath,et al.  Learning Raw Image Reconstruction-Aware Deep Image Compressors , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Xiaogang Wang,et al.  StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Jiro Katto,et al.  Energy Compaction-Based Image Compression Using Convolutional AutoEncoder , 2020, IEEE Transactions on Multimedia.

[57]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[58]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Chia-Yen Lee,et al.  Learning to Detect Fake Face Images in the Wild , 2018, 2018 International Symposium on Computer, Consumer and Control (IS3C).

[60]  Shuang Wang,et al.  Fuzzy Sparse Autoencoder Framework for Single Image Per Person Face Recognition , 2018, IEEE Transactions on Cybernetics.

[61]  Geoffrey E. Hinton,et al.  Transforming Auto-Encoders , 2011, ICANN.

[62]  Robert Chesney,et al.  Deepfakes and the New Disinformation War , 2018 .

[63]  Yao Zhao,et al.  Recapture Image Forensics Based on Laplacian Convolutional Neural Networks , 2016, IWDW.

[64]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[65]  Robert Chesney,et al.  Disinformation on Steroids: The Threat of Deep Fakes , 2018 .

[66]  Khaled Salah,et al.  Combating Deepfake Videos Using Blockchain and Smart Contracts , 2019, IEEE Access.

[67]  Premkumar Natarajan,et al.  Recurrent Convolutional Strategies for Face Manipulation Detection in Videos , 2019, CVPR Workshops.

[68]  Stefan Duffner,et al.  Siamese multi-layer perceptrons for dimensionality reduction and face identification , 2015, Multimedia Tools and Applications.

[69]  L. Floridi.  Artificial Intelligence, Deepfakes and a Future of Ectypes , 2018, Philosophy & Technology.

[70]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Ueli Maurer,et al.  Authentication theory and hypothesis testing , 2000, IEEE Trans. Inf. Theory.

[72]  Lucas Theis,et al.  Fast Face-Swap Using Convolutional Neural Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[73]  Yann LeCun,et al.  Learning a similarity metric discriminatively, with application to face verification , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[74]  Andreas Rössler,et al.  FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces , 2018, ArXiv.

[75]  Matthieu Cord,et al.  Gaze latent support vector machine for image classification , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[76]  Robert M. Chesney,et al.  Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security , 2018 .

[77]  Larry S. Davis,et al.  Two-Stream Neural Networks for Tampered Face Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[78]  Victor Lempitsky,et al.  Few-Shot Adversarial Learning of Realistic Neural Talking Head Models , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[79]  Shuang Bai,et al.  Growing random forest on deep convolutional neural networks for scene categorization , 2017, Expert Syst. Appl..