Beyond Pixels: Image Provenance Analysis Leveraging Metadata

Creative works, whether paintings or memes, follow unique journeys that result in their final form. Understanding these journeys, a process known as "provenance analysis," provides rich insights into the use, motivation, and authenticity underlying any given work. The application of this type of study to the expanse of unregulated content on the Internet is what we consider in this paper. Provenance analysis provides a snapshot of the chronology and validity of content as it is uploaded, re-uploaded, and modified over time. Although still in its infancy, automated provenance analysis for online multimedia is already being applied to different types of content. Most current works seek to build provenance graphs based on the shared content between images or videos. This can be a computationally expensive task, especially when considering the vast influx of content that the Internet sees every day. Utilizing non-content-based information, such as timestamps, geotags, and camera IDs can help provide important insights into the path a particular image or video has traveled during its time on the Internet without large computational overhead. This paper tests the scope and applicability of metadata-based inferences for provenance graph construction in two different scenarios: digital image forensics and cultural analytics.

[1]  Lamberto Ballan,et al.  Love Thy Neighbors: Image Annotation by Exploiting Image Metadata , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Silvana Toffolo,et al.  Evaluation Plan , 2014 .

[3]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Babak Mahdian,et al.  A bibliography on blind methods for identifying image forgery , 2010, Signal Process. Image Commun..

[5]  Edward Y. Chang,et al.  Extent: Inferring Image Metadata from Context and Content , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[6]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[7]  James A. Hendler,et al.  Annotation and Provenance Tracking in Semantic Web Photo Libraries , 2006, IPAW.

[8]  Yan Ke,et al.  Efficient Near-duplicate Detection and Sub-image Retrieval , 2004 .

[9]  Anderson Rocha,et al.  Image Phylogeny by Minimal Spanning Trees , 2012, IEEE Transactions on Information Forensics and Security.

[10]  Alex ChiChung Kot,et al.  Blurred Image Splicing Localization by Exposing Blur Type Inconsistency , 2015, IEEE Transactions on Information Forensics and Security.

[11]  Hsiang-Cheh Huang,et al.  Metadata-based image watermarking for copyright protection , 2010, Simul. Model. Pract. Theory.

[12]  Davide Cozzolino,et al.  Attacking the triangle test in sensor-based camera identification , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[13]  Kevin Li,et al.  Faceted metadata for image search and browsing , 2003, CHI '03.

[14]  J. Kruskal On the shortest spanning subtree of a graph and the traveling salesman problem , 1956 .

[15]  Andrew Zisserman,et al.  Near Duplicate Image Detection: min-Hash and tf-idf Weighting , 2008, BMVC.

[16]  Hany Farid,et al.  Digital Image Authentication From JPEG Headers , 2011, IEEE Transactions on Information Forensics and Security.

[17]  榎本 聖一 パリエッフェル塔 = Paris-The Eiffel Tower , 2014 .

[18]  Alex ChiChung Kot,et al.  Estimating EXIF Parameters Based on Noise Features for Image Manipulation Detection , 2013, IEEE Transactions on Information Forensics and Security.

[19]  Jake K. Aggarwal,et al.  Applying perceptual grouping to content-based image retrieval: building images , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[20]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[21]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Paul Alvarez,et al.  Using Extended File Information (EXIF) File Headers in Digital Evidence Analysis , 2004, Int. J. Digit. EVid..

[23]  Mauro Barni,et al.  Multiple parenting identification in image phylogeny , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[24]  Patrick J. Flynn,et al.  Image Provenance Analysis at Scale , 2018, IEEE Transactions on Image Processing.

[25]  Sanjeev Khanna,et al.  Why and Where: A Characterization of Data Provenance , 2001, ICDT.

[26]  Jiri Matas,et al.  Robust wide-baseline stereo from maximally stable extremal regions , 2004, Image Vis. Comput..

[27]  Christos Faloutsos,et al.  Automatic image captioning , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[28]  Can Chen,et al.  Image Splicing Detection via Camera Response Function Analysis , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Jiwu Huang,et al.  Anti-Forensics of Camera Identification and the Triangle Test by Improved Fingerprint-Copy Attack , 2017, ArXiv.

[30]  Bertram Ludäscher,et al.  Provenance browser: Displaying and querying scientific workflow provenance graphs , 2010, 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010).

[31]  S. Goldenstein,et al.  Toward image phylogeny forests: automatically recovering semantically similar image relationships. , 2013, Forensic science international.

[32]  Xiangyu Zhang,et al.  ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  James Cheney,et al.  A Graph Model of Data and Workflow Provenance , 2010, TaPP.

[34]  Anderson Rocha,et al.  U-Phylogeny: Undirected provenance graph construction in the wild , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[35]  Vijay H. Mankar,et al.  Digital image forgery detection using passive techniques: A survey , 2013, Digit. Investig..

[36]  Mark J. Huiskes,et al.  The MIR flickr retrieval evaluation , 2008, MIR '08.

[37]  Paolo Bestagini,et al.  Spotting the difference: Context retrieval and analysis for improved forgery detection and localization , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[38]  Jiebo Luo,et al.  Photo classification by integrating image content and camera metadata , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[39]  Hayit Greenspan,et al.  Content-Based Image Retrieval in Radiology: Current Status and Future Directions , 2010, Journal of Digital Imaging.

[40]  Mauro Barni,et al.  Multiple Parenting Phylogeny Relationships in Digital Images , 2016, IEEE Transactions on Information Forensics and Security.

[41]  Alessandro Piva,et al.  Image splicing detection based on general perspective constraints , 2015, 2015 IEEE International Workshop on Information Forensics and Security (WIFS).

[42]  Davide Cozzolino,et al.  Splicebuster: A new blind image splicing detector , 2015, 2015 IEEE International Workshop on Information Forensics and Security (WIFS).

[43]  Yogesh L. Simmhan,et al.  A survey of data provenance techniques , 2005 .

[44]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[45]  Cordelia Schmid,et al.  Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search , 2008, ECCV.

[46]  S. Sasikala,et al.  Efficient Content Based Image Retrieval System with Metadata Processing , 2015 .

[47]  Anderson Rocha,et al.  Provenance filtering for multimedia phylogeny , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[48]  Andrew Owens,et al.  Fighting Fake News: Image Splice Detection via Learned Self-Consistency , 2018, ECCV.

[49]  Hae-Yeoun Lee,et al.  Estimation of color modification in digital images by CFA pattern change. , 2013, Forensic science international.

[50]  H. Farid,et al.  Image forgery detection , 2009, IEEE Signal Processing Magazine.