Media Forensics Considerations on DeepFake Detection with Hand-Crafted Features

DeepFake detection is a novel task for media forensics and is currently receiving a lot of research attention due to the threat these targeted video manipulations propose to the trust placed in video footage. The current trend in DeepFake detection is the application of neural networks to learn feature spaces that allow them to be distinguished from unmanipulated videos. In this paper, we discuss, with features hand-crafted by domain experts, an alternative to this trend. The main advantage that hand-crafted features have over learned features is their interpretability and the consequences this might have for plausibility validation for decisions made. Here, we discuss three sets of hand-crafted features and three different fusion strategies to implement DeepFake detection. Our tests on three pre-existing reference databases show detection performances that are under comparable test conditions (peak AUC > 0.95) to those of state-of-the-art methods using learned features. Furthermore, our approach shows a similar, if not better, generalization behavior than neural network-based methods in tests performed with different training and test sets. In addition to these pattern recognition considerations, first steps of a projection onto a data-centric examination approach for forensics process modeling are taken to increase the maturity of the present investigation.

[1]  Lisa M. DeBruine,et al.  Face Research Lab London Set , 2017 .

[2]  Anil K. Jain,et al.  Simultaneous feature selection and clustering using mixture models , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Siwei Lyu,et al.  Exposing DeepFake Videos By Detecting Face Warping Artifacts , 2018, CVPR Workshops.

[4]  Ilke Demir,et al.  FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals , 2019, IEEE transactions on pattern analysis and machine intelligence.

[5]  Stefanos Zafeiriou,et al.  300 Faces In-The-Wild Challenge: database and results , 2016, Image Vis. Comput..

[6]  Robert Chesney,et al.  Deepfakes and the New Disinformation War , 2018 .

[7]  Jana Dittmann,et al.  Modeling Attacks on Photo-ID Documents and Applying Media Forensics for the Detection of Facial Morphing , 2017, IH&MMSec.

[8]  Andrew Chadwick,et al.  Deepfakes and Disinformation: Exploring the Impact of Synthetic Political Video on Deception, Uncertainty, and Trust in News , 2020, Social Media + Society.

[9]  Andreas Rössler,et al.  FaceForensics++: Learning to Detect Manipulated Facial Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Sébastien Marcel,et al.  DeepFakes: a New Threat to Face Recognition? Assessment and Detection , 2018, ArXiv.

[11]  Jana Dittmann,et al.  A new forensic model and its application to the collection, extraction and long term storage of screen content off a memory dump , 2009, 2009 16th International Conference on Digital Signal Processing.

[12]  Christian Riess,et al.  Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations , 2019, 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[13]  N. Krawetz.  A Picture ’ s Worth . . . Digital Image Analysis and Forensics Version 2 , 2007 .

[14]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[15]  R. Altschaffel.  Computer forensics in cyber-physical systems: applying existing forensic knowledge and procedures from classical IT to automation and automotive , 2020 .

[16]  Alexander Binder,et al.  Evaluating the Visualization of What a Deep Neural Network Has Learned , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[17]  Andreas Rössler,et al.  FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces , 2018, ArXiv.

[18]  Aythami Morales,et al.  DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection , 2020, Inf. Fusion.

[19]  Siwei Lyu,et al.  In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking , 2018, ArXiv.

[20]  Wenyi Lin,et al.  Comparison of handcrafted features and convolutional neural networks for liver MR image adequacy assessment , 2020, Scientific Reports.

[21]  Keecheon Kim,et al.  DeepVision: Deepfakes Detection Using Human Eye Blinking Pattern , 2020, IEEE Access.

[22]  Amparo Alonso-Betanzos,et al.  Filter Methods for Feature Selection - A Comparative Study , 2007, IDEAL.

[23]  Christian Krätzer.  Statistical pattern recognition for audio-forensics : empirical investigations on the application scenarios audio steganalysis and microphone forensics , 2013 .

[24]  Worku Muluye Wubet.  The Deepfake Challenges and Deepfake Video Detection , 2020 .

[25]  Luisa Verdoliva,et al.  Media Forensics and DeepFakes: An Overview , 2020, IEEE Journal of Selected Topics in Signal Processing.

[26]  Weiguo Zhang,et al.  A Novel Counterfeit Feature Extraction Technique for Exposing Face-Swap Images Based on Deep Learning and Error Level Analysis , 2020, Entropy.

[27]  Saeid Nahavandi,et al.  Deep Learning for Deepfakes Creation and Detection , 2019, ArXiv.

[28]  Felix C. Freiling,et al.  Multimedia Forensics Is Not Computer Forensics , 2009, IWCF.

[29]  Xin Yang,et al.  Exposing Deep Fakes Using Inconsistent Head Poses , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Weiguo Zhang,et al.  Exposing Face-Swap Images Based on Deep Learning and ELA Detection , 2019 .

[31]  Hao Li,et al.  Protecting World Leaders Against Deep Fakes , 2019, CVPR Workshops.

[32]  Jianwei Fei,et al.  A Survey on Deepfake Video Detection , 2021, IET Biom..

[33]  Brian C. Lovell,et al.  Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference , 2009, ICB.

[34]  Christophe Champod,et al.  Scientific Evidence in Europe -- Admissibility, Evaluation and Equality of Arms , 2011 .

[35]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[36]  Jana Dittmann,et al.  Automatic Generation and Detection of Visually Faultless Facial Morphs , 2017, VISIGRAPP.

[37]  Jana Dittmann,et al.  Supporting Forensic Design - A Course Profile to Teach Forensics , 2015, 2015 Ninth International Conference on IT Security Incident Management & IT Forensics.

[38]  Stefan Kiltz,et al.  Data-Centric Examination Approach (DCEA) for a qualitative determination of error, loss and uncertainty in digital and digitised forensics , 2020 .

[39]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[40]  Scott McCloskey,et al.  Detecting GAN-generated Imagery using Color Cues , 2018, ArXiv.

[41]  Alexander Binder,et al.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation , 2015, PloS one.

[42]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[43]  Premkumar Natarajan,et al.  Recurrent Convolutional Strategies for Face Manipulation Detection in Videos , 2019, CVPR Workshops.

[44]  Paolo Bestagini,et al.  Training Strategies and Data Augmentations in CNN-based DeepFake Video Detection , 2020, 2020 IEEE International Workshop on Information Forensics and Security (WIFS).

[45]  Maneesh Agrawala,et al.  Detecting Deep-Fake Videos from Phoneme-Viseme Mismatches , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[46]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .