Deep Learning for Deepfakes Creation and Detection

Deep learning has been successfully applied to solve various complex problems ranging from big data analytics to computer vision and human-level control. Deep learning advances however have also been employed to create software that can cause threats to privacy, democracy and national security. One of those deep learning-powered applications recently emerged is “deepfake”. Deepfake algorithms can create fake images and videos that humans cannot distinguish them from authentic ones. The proposal of technologies that can automatically detect and assess the integrity of digital visual media is therefore indispensable. This paper presents a survey of algorithms used to create deepfakes and, more importantly, methods proposed to detect deepfakes in the literature to date. We present extensive discussions on challenges, research trends and directions related to deepfake technologies. By reviewing the background of deepfakes and state-of-the-art deepfake detection methods, this study provides a comprehensive overview of deepfake techniques and facilitates the development of new and more robust methods to deal with the increasingly challenging deepfakes.

[1]  Marie-Helen Maras,et al.  Determining authenticity of video evidence in the age of artificial intelligence and in the wake of Deepfake videos , 2018, The International Journal of Evidence & Proof.

[2]  Edward J. Delp,et al.  Deepfake Video Detection Using Recurrent Neural Networks , 2018, 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[3]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  H. Farid,et al.  Image forgery detection , 2009, IEEE Signal Processing Magazine.

[5]  Premkumar Natarajan,et al.  Recurrent Convolutional Strategies for Face Manipulation Detection in Videos , 2019, CVPR Workshops.

[6]  Yao Zhao,et al.  Recapture Image Forensics Based on Laplacian Convolutional Neural Networks , 2016, IWDW.

[7]  Andreas Rössler,et al.  FaceForensics++: Learning to Detect Manipulated Facial Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Lucas Theis,et al.  Fast Face-Swap Using Convolutional Neural Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[9]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Junichi Yamagishi,et al.  Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Qingmin Liao,et al.  FV-GAN: Finger Vein Representation Using Generative Adversarial Networks , 2019, IEEE Transactions on Information Forensics and Security.

[12]  Siwei Lyu,et al.  In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking , 2018, 2018 IEEE International Workshop on Information Forensics and Security (WIFS).

[13]  Geoffrey E. Hinton,et al.  Transforming Auto-Encoders , 2011, ICANN.

[14]  Raymond Y. K. Lau,et al.  Least Squares Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[15]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[16]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[17]  Xiaogang Wang,et al.  StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[19]  Junichi Yamagishi,et al.  Distinguishing computer graphics from natural images using convolution neural networks , 2017, 2017 IEEE Workshop on Information Forensics and Security (WIFS).

[20]  Ueli Maurer,et al.  Authentication theory and hypothesis testing , 2000, IEEE Trans. Inf. Theory.

[21]  Jianmei Yang,et al.  A Fast Forgery Detection Algorithm Based on Exponential-Fourier Moments for Video Region Duplication , 2018, IEEE Transactions on Multimedia.

[22]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[23]  Belhassen Bayar,et al.  A Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer , 2016, IH&MMSec.

[24]  Sébastien Marcel,et al.  Face Anti-spoofing Based on General Image Quality Assessment , 2014, 2014 22nd International Conference on Pattern Recognition.

[25]  Victor Lempitsky,et al.  Few-Shot Adversarial Learning of Realistic Neural Talking Head Models , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[26]  Jiro Katto,et al.  Energy Compaction-Based Image Compression Using Convolutional AutoEncoder , 2020, IEEE Transactions on Multimedia.

[27]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Marco Fontani,et al.  A Video Forensic Framework for the Unsupervised Analysis of MP4-Like File Container , 2019, IEEE Transactions on Information Forensics and Security.

[29]  Jing Dong,et al.  Deep learning for steganalysis via convolutional neural networks , 2015, Electronic Imaging.

[30]  Jing Dong,et al.  On the generalization of GAN image forensics , 2019, CCBR.

[31]  Pablo Garrido,et al.  High-Fidelity Monocular Face Reconstruction Based on an Unsupervised Model-Based Face Autoencoder , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Chih-Chung Hsu,et al.  Deep Fake Image Detection Based on Pairwise Learning , 2020, Applied Sciences.

[33]  Bolin Chen,et al.  Fake Faces Identification via Convolutional Neural Network , 2018, IH&MMSec.

[34]  Khaled Salah,et al.  Combating Deepfake Videos Using Blockchain and Smart Contracts , 2019, IEEE Access.

[35]  Larry S. Davis,et al.  Two-Stream Neural Networks for Tampered Face Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[36]  Ira Kemelmacher-Shlizerman,et al.  Synthesizing Obama , 2017, ACM Trans. Graph..

[37]  Ying Zhang,et al.  Automated face swapping and its detection , 2017, 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP).

[38]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[39]  Matthieu Cord,et al.  Gaze latent support vector machine for image classification , 2016, ICIP.

[40]  Joon Son Chung,et al.  Lip Reading Sentences in the Wild , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[42]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[43]  Jessica J. Fridrich,et al.  Rich Models for Steganalysis of Digital Images , 2012, IEEE Transactions on Information Forensics and Security.

[44]  Christian Riess,et al.  Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations , 2019, 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[45]  Zhenan Sun,et al.  3D Aided Duet GANs for Multi-View Face Image Synthesis , 2019, IEEE Transactions on Information Forensics and Security.

[46]  Sébastien Marcel,et al.  Vulnerability assessment and detection of Deepfake videos , 2019, 2019 International Conference on Biometrics (ICB).

[47]  Fang Liu,et al.  Task-Oriented GAN for PolSAR Image Classification and Clustering , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[49]  Chia-Yen Lee,et al.  Learning to Detect Fake Face Images in the Wild , 2018, 2018 International Symposium on Computer, Consumer and Control (IS3C).

[50]  Abhijith Punnappurath,et al.  Learning Raw Image Reconstruction-Aware Deep Image Compressors , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Xin Yang,et al.  Exposing Deep Fakes Using Inconsistent Head Poses , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[52]  Yann LeCun,et al.  Learning a similarity metric discriminatively, with application to face verification , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[53]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[54]  Shuang Bai,et al.  Growing random forest on deep convolutional neural networks for scene categorization , 2017, Expert Syst. Appl..

[55]  Amy N. Yates,et al.  MFC Datasets: Large-Scale Benchmark Datasets for Media Forensic Challenge Evaluation , 2019, 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[56]  Justus Thies,et al.  Face2Face: real-time face capture and reenactment of RGB videos , 2019, Commun. ACM.

[57]  Stefan Duffner,et al.  Siamese multi-layer perceptrons for dimensionality reduction and face identification , 2015, Multimedia Tools and Applications.

[58]  Andreas Rössler,et al.  FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces , 2018, ArXiv.

[59]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[60]  Robert M. Chesney,et al.  Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security , 2018 .

[61]  Miroslav Goljan,et al.  Digital camera identification from sensor pattern noise , 2006, IEEE Transactions on Information Forensics and Security.

[62]  Junichi Yamagishi,et al.  MesoNet: a Compact Facial Video Forgery Detection Network , 2018, 2018 IEEE International Workshop on Information Forensics and Security (WIFS).

[63]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[65]  Sébastien Marcel,et al.  On the effectiveness of local binary patterns in face anti-spoofing , 2012, 2012 BIOSIG - Proceedings of the International Conference of Biometrics Special Interest Group (BIOSIG).

[66]  L. Floridi.  Artificial Intelligence, Deepfakes and a Future of Ectypes , 2018, Philosophy & Technology.

[67]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[68]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Robert Chesney,et al.  Deepfakes and the New Disinformation War , 2018 .

[70]  Husrev T. Sencar,et al.  A study of the robustness of PRNU-based camera identification , 2009, Electronic Imaging.

[71]  Davide Cozzolino,et al.  Detection of GAN-Generated Fake Images over Social Networks , 2018, 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[72]  Anil K. Jain,et al.  Face Spoof Detection With Image Distortion Analysis , 2015, IEEE Transactions on Information Forensics and Security.

[73]  Sakshi Agarwal,et al.  Limits of Deepfake Detection: A Robust Estimation Viewpoint , 2019, ArXiv.

[74]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[75]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[76]  Sébastien Marcel,et al.  Speaker Inconsistency Detection in Tampered Video , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[77]  Siwei Lyu,et al.  Exposing DeepFake Videos By Detecting Face Warping Artifacts , 2018, CVPR Workshops.

[78]  Shuang Wang,et al.  Fuzzy Sparse Autoencoder Framework for Single Image Per Person Face Recognition , 2018, IEEE Transactions on Cybernetics.

[79]  Ron J. Weiss,et al.  Unsupervised Speech Representation Learning Using WaveNet Autoencoders , 2019, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[80]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[81]  Robert Chesney,et al.  Disinformation on Steroids: The Threat of Deep Fakes , 2018 .