Process-Driven Modelling of Media Forensic Investigations-Considerations on the Example of DeepFake Detection

Academic research in media forensics mainly focuses on methods for the detection of the traces or artefacts left by media manipulations in media objects. While the resulting detectors often achieve quite impressive detection performances, when tested under lab conditions, hardly any of those have yet come close to the ultimate benchmark for any forensic method, which would be courtroom readiness. This paper tries first to facilitate the different stakeholder perspectives in this field and then to partly address the apparent gap between the academic research community and the requirements imposed onto forensic practitioners. The intention is to facilitate the mutual understanding of these two classes of stakeholders and assist with first steps intended at closing this gap. To do so, first a concept for modelling media forensic investigation pipelines is derived from established guidelines. Then, the applicability of such modelling is illustrated on the example of a fusion-based media forensic investigation pipeline aimed at the detection of DeepFake videos using five exemplary detectors (hand-crafted, in one case neural network supported) and testing two different fusion operators. At the end of the paper, the benefits of such a planned realisation of AI-based investigation methods are discussed and generalising effects are mapped out.

[1]  A. Torralba,et al.  Natural Language Descriptions of Deep Visual Features , 2022, ICLR.

[2]  Siegel Dennis,et al.  Forensic data model for artificial intelligence based media forensics - Illustrated on the example of DeepFake detection , 2022, Electronic Imaging.

[3]  Saeid Nahavandi,et al.  Deep learning for deepfakes creation and detection: A survey , 2019, Comput. Vis. Image Underst..

[4]  Handbook of Digital Face Manipulation and Detection: From DeepFakes to Morphing Attacks , 2022, Handbook of Digital Face Manipulation and Detection.

[5]  Anubhav Jain,et al.  Improving Generalization of Deepfake Detection by Training for Attribution , 2021, 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP).

[6]  Chang Xu,et al.  DeepFake MNIST+: A DeepFake Facial Animation Dataset , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[7]  Simon S. Woo,et al.  FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset , 2021, NeurIPS Datasets and Benchmarks.

[8]  Potential advantages and limitations of using information fusion in media forensics—a discussion on the example of detecting face morphing attacks , 2021, EURASIP J. Inf. Secur..

[9]  Jana Dittmann,et al.  Media Forensics Considerations on DeepFake Detection with Hand-Crafted Features , 2021, J. Imaging.

[10]  Jaeseong You,et al.  KoDF: A Large-scale Korean DeepFake Detection Dataset , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Yisroel Mirsky,et al.  The Creation and Detection of Deepfakes , 2020, ACM Comput. Surv..

[12]  Yu-Gang Jiang,et al.  WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection , 2020, ACM Multimedia.

[13]  Brian Dolhansky,et al.  The DeepFake Detection Challenge Dataset , 2020, ArXiv.

[14]  Keecheon Kim,et al.  DeepVision: Deepfakes Detection Using Human Eye Blinking Pattern , 2020, IEEE Access.

[15]  Chen Change Loy,et al.  DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Siwei Lyu,et al.  Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Stefan Kiltz,et al.  Data-Centric Examination Approach (DCEA) for a qualitative determination of error, loss and uncertainty in digital and digitised forensics , 2020 .

[18]  R. Altschaffel.  Computer forensics in cyber-physical systems: applying existing forensic knowledge and procedures from classical IT to automation and automotive , 2020 .

[19]  Cristian Canton-Ferrer,et al.  The Deepfake Detection Challenge (DFDC) Preview Dataset , 2019, ArXiv.

[20]  Andreas Rössler,et al.  FaceForensics++: Learning to Detect Manipulated Facial Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  Xin Yang,et al.  Exposing Deep Fakes Using Inconsistent Head Poses , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Hao Li,et al.  Protecting World Leaders Against Deep Fakes , 2019, CVPR Workshops.

[23]  Sébastien Marcel,et al.  DeepFakes: a New Threat to Face Recognition? Assessment and Detection , 2018, ArXiv.

[24]  Scott McCloskey,et al.  Detecting GAN-generated Imagery using Color Cues , 2018, ArXiv.

[25]  Joon Son Chung,et al.  VoxCeleb2: Deep Speaker Recognition , 2018, INTERSPEECH.

[26]  Siwei Lyu,et al.  In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking , 2018, ArXiv.

[27]  Andreas Rössler,et al.  FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces , 2018, ArXiv.

[28]  Xiangliang Zhang,et al.  An up-to-date comparison of state-of-the-art classification algorithms , 2017, Expert Syst. Appl..

[29]  Lucas Theis,et al.  Fast Face-Swap Using Convolutional Neural Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Katharina Wagner,et al.  Digital Evidence And Computer Crime Forensic Science Computers And The Internet , 2016 .

[31]  Jan Cech,et al.  Real-Time Eye Blink Detection using Facial Landmarks , 2016 .

[32]  Shujun Li,et al.  Handbook of Digital Forensics of Multimedia Data and Devices , 2015 .

[33]  Jana Dittmann,et al.  Supporting Forensic Design - A Course Profile to Teach Forensics , 2015, 2015 Ninth International Conference on IT Security Incident Management & IT Forensics.

[34]  P. Bartholomew.  Seize First, Search Later: The Hunt for Digital Evidence , 2014 .

[35]  Sarah V. Hart,et al.  Forensic Examination of Digital Evidence: A Guide for Law Enforcement , 2014 .

[36]  Christian Krätzer.  Statistical pattern recognition for audio-forensics : empirical investigations on the application scenarios audio steganalysis and microphone forensics , 2013 .

[37]  Matti Pietikäinen,et al.  Bi-Modal Person Recognition on a Mobile Phone: Using Mobile Phone Data , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[38]  Christophe Champod,et al.  Scientific Evidence in Europe -- Admissibility, Evaluation and Equality of Arms , 2011 .

[39]  Larry E. Daniel,et al.  Digital Forensics for Legal Professionals: Understanding Digital Evidence from the Warrant to the Courtroom , 2011 .

[40]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[41]  Brian Reichow,et al.  Evidence-based practices and treatments for children with autism , 2011 .

[42]  N. Daéid,et al.  Interpol's Forensic Science Review , 2010 .

[43]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[44]  Brian C. Lovell,et al.  Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference , 2009, ICB.

[45]  Jessica J. Fridrich,et al.  Large scale test of sensor fingerprint camera identification , 2009, Electronic Imaging.

[46]  Zeno Geradts,et al.  Forensic audio and visual evidence 2004-2007 : A Review , 2007 .

[47]  Eibe Frank,et al.  Speeding Up Logistic Model Tree Induction , 2005, PKDD.

[48]  J. Sim,et al.  The kappa statistic in reliability studies: use, interpretation, and sample size requirements. , 2005, Physical therapy.

[49]  Eibe Frank,et al.  Logistic Model Trees , 2003, Machine Learning.

[50]  Barbara Maria Di Eugenio,et al.  Squibs and Discussions - The Kappa Statistic , 2004 .

[51]  Matthew Meyers,et al.  Computer Forensics: The Need for Standardization and Certification , 2004, Int. J. Digit. EVid..

[52]  Bill Nelson,et al.  Guide to Computer Forensics and Investigations , 2003 .

[53]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[54]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[55]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[56]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[57]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .