Discovering Image Manipulation History by Pairwise Relation and Forensics Tools

Given a potentially manipulated probe image, provenance analysis aims to find all images derived from the probe (offsprings) and all images from which the probe is derived (ancestors) in a large dataset (provenance filtering), and reconstruct the manipulation history with the retrieved images (provenance graph building). In this paper, we address two major challenges in provenance analysis, retrieving the source image of the small regions that are spliced into the probe image, and, detecting source images within the search results. For the former challenge, we propose to detect spliced regions by pairwise image comparison and only use local features extracted from the spliced region to perform an additional search. This removes the influence of the background and greatly improves the recall. For the latter, we propose to learn a pairwise ancestor-offspring detector and use it jointly with a holistic image manipulation detector to identify the source image. The proposed provenance analysis system has performed remarkably in evaluations using comprehensive provenance datasets. It's the winning solution for NIST Media Forensics Challenge (MFC) in 2018, 2019 and 2020. In MFC 2019, our provenance results achieved a 12% improvement in filtering and a 20% gain in oracle provenance graphs building over the alternative methods. In the real-world Reddit dataset, the edge overlap between our reconstructed provenance graphs and the ground-truth graphs is 5 times better than the state-of-the-art system.

[1]  Michael Isard,et al.  Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[2]  Anderson Rocha,et al.  U-Phylogeny: Undirected provenance graph construction in the wild , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[3]  Noel E. O'Connor,et al.  Bags of Local Convolutional Features for Scalable Instance Search , 2016, ICMR.

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Jian Sun,et al.  Optimized Product Quantization for Approximate Nearest Neighbor Search , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Victor S. Lempitsky,et al.  Efficient Indexing of Billion-Scale Datasets of Deep Descriptors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Hany Farid,et al.  Photo forensics from JPEG dimples , 2017, 2017 IEEE Workshop on Information Forensics and Security (WIFS).

[8]  Anderson Rocha,et al.  Beyond Pixels: Image Provenance Analysis Leveraging Metadata , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[9]  Cordelia Schmid,et al.  Product Quantization for Nearest Neighbor Search , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  H. Farid.  Photo Forensics , 2016 .

[11]  Atsuto Maki,et al.  A Baseline for Visual Instance Retrieval with Deep Convolutional Networks , 2014, ICLR 2015.

[12]  H. Farid,et al.  Image forgery detection , 2009, IEEE Signal Processing Magazine.

[13]  Anderson Rocha,et al.  Exploring heuristic and optimum branching algorithms for image phylogeny , 2013, J. Vis. Commun. Image Represent..

[14]  Bin Fan,et al.  L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Anderson Rocha,et al.  Video Phylogeny: Recovering near-duplicate video relationships , 2011, 2011 IEEE International Workshop on Information Forensics and Security.

[16]  John R. Kender,et al.  Visual memes in social media: tracking real-world news in YouTube videos , 2011, ACM Multimedia.

[17]  Jeff Johnson,et al.  Billion-Scale Similarity Search with GPUs , 2017, IEEE Transactions on Big Data.

[18]  Honggang Qi,et al.  Contrast Enhancement Estimation for Digital Image Forensics , 2017, ACM Trans. Multim. Comput. Commun. Appl..

[19]  Anderson Rocha,et al.  Provenance filtering for multimedia phylogeny , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[20]  Tiberio Uricchio,et al.  Localization of JPEG Double Compression Through Multi-domain Convolutional Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Walter J. Scheirer,et al.  Practical Text Phylogeny for Real-World Settings , 2018, IEEE Access.

[22]  Anindya Sarkar,et al.  Detection of seam carving and localization of seam insertions in digital images , 2009, MM&Sec '09.

[23]  S. Goldenstein,et al.  Toward image phylogeny forests: automatically recovering semantically similar image relationships. , 2013, Forensic science international.

[24]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[25]  Anthony Hoogs,et al.  Object insertion and removal in images with mirror reflection , 2017, 2017 IEEE Workshop on Information Forensics and Security (WIFS).

[26]  Yue Wu,et al.  Deep Multimodal Image-Repurposing Detection , 2018, ACM Multimedia.

[27]  Anderson Rocha,et al.  Image Phylogeny by Minimal Spanning Trees , 2012, IEEE Transactions on Information Forensics and Security.

[28]  Davide Cozzolino,et al.  Splicebuster: A new blind image splicing detector , 2015, 2015 IEEE International Workshop on Information Forensics and Security (WIFS).

[29]  Mauro Barni,et al.  Multiple parenting identification in image phylogeny , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[30]  Patrick J. Flynn,et al.  Image Provenance Analysis at Scale , 2018, IEEE Transactions on Image Processing.

[31]  Hervé Jégou,et al.  Visual query expansion with or without geometry: Refining local descriptors by feature aggregation , 2014, Pattern Recognit..

[32]  Hany Farid,et al.  Rebroadcast Attacks: Defenses, Reattacks, and Redefenses , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[33]  Anderson Rocha,et al.  New dissimilarity measures for image phylogeny reconstruction , 2017, Pattern Analysis and Applications.

[34]  Shih-Fu Chang,et al.  Internet image archaeology: automatically tracing the manipulation history of photographs on the web , 2008, ACM Multimedia.

[35]  Jonathan G. Fiscus,et al.  MFC Datasets: Large-Scale Benchmark Datasets for Media Forensic Challenge Evaluation , 2019, 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[36]  Krystian Mikolajczyk,et al.  Learning local feature descriptors with triplets and shallow convolutional neural networks , 2016, BMVC.

[37]  Wael Abd-Almageed,et al.  BusterNet: Detecting Copy-Move Image Forgery with Source/Target Localization , 2018, ECCV.

[38]  Davide Cozzolino,et al.  Detection of GAN-Generated Fake Images over Social Networks , 2018, 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[39]  Mauro Barni,et al.  Multiple Parenting Phylogeny Relationships in Digital Images , 2016, IEEE Transactions on Information Forensics and Security.

[40]  Ondrej Chum,et al.  CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples , 2016, ECCV.

[41]  Amy N. Yates,et al.  2018 MediFor Challenge , 2019 .