Identifying Cross-Depicted Historical Motifs

Cross-depiction is the problem of identifying the same object even when it is depicted in a variety of manners.This is a common problem in handwritten historical document image analysis, for instance when the same letter or motif is depicted in several different ways. It is a simple task for humans yet conventional computer vision methods struggle to cope with it. In this paper we address this problem using state-of-the-art deep learning techniques on a dataset of historical watermarks containing images created with different methods of reproduction, such as hand tracing, rubbing, and radiography.To study the robustness of deep learning based approaches to the cross-depiction problem, we measure their performance on two different tasks: classification and similarity rankings. For the former we achieve a classification accuracy of 96 % using deep convolutional neural networks. For the latter we have a false positive rate at 95% recall of 0.11. These results outperform state-of-the-art methods by a significant margin.

[1]  Rahul Sukthankar,et al.  MatchNet: Unifying feature and metric learning for patch-based matching , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  W. H. Ittelson,et al.  Experiments in Perception , 1951 .

[4]  John P. Eakins,et al.  Content-Based Retrieval of Historical Watermark Images: I-tracings , 2002, CIVR.

[5]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[7]  Lambert Schomaker,et al.  Historical manuscript dating based on temporal pattern codebook , 2016, Comput. Vis. Image Underst..

[8]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[9]  Anders Brun,et al.  Large scale style based dating of medieval manuscripts , 2015, HIP@ICDAR.

[10]  Dawn Song,et al.  Robust Physical-World Attacks on Deep Learning Models , 2017, 1707.08945.

[11]  Daniel Manger,et al.  Large-Scale Tattoo Image Retrieval , 2012, 2012 Ninth Conference on Computer and Robot Vision.

[12]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  David Picard,et al.  Non-negative dictionary learning for paper watermark similarity , 2016, 2016 50th Asilomar Conference on Signals, Systems and Computers.

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Krystian Mikolajczyk,et al.  Learning local feature descriptors with triplets and shallow convolutional neural networks , 2016, BMVC.

[17]  Gerd Brunner,et al.  Structure features for content based image retrieval and classification problems , 2007 .

[18]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  John P. Eakins,et al.  Content-Based Retrieval of Historical Watermark Images: II - Electron Radiographs , 2003, CIVR.

[20]  Rui Hu,et al.  A performance evaluation of gradient field HOG descriptor for sketch based image retrieval , 2013, Comput. Vis. Image Underst..

[21]  Marcus Liwicki,et al.  DeepDIVA: A Highly-Functional Python Framework for Reproducible Experiments , 2018, 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR).

[22]  Mathias Lux,et al.  Content based image retrieval with LIRe , 2011, ACM Multimedia.

[23]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[24]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[25]  Nir Ailon,et al.  Deep Metric Learning Using Triplet Network , 2014, SIMBAD.

[26]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Thierry Pun,et al.  Archival and retrieval of historical watermark images , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[28]  Jitendra Malik,et al.  Detecting People in Cubist Art , 2014, ECCV Workshops.

[29]  Yang Song,et al.  Learning Fine-Grained Image Similarity with Deep Ranking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Marcus Liwicki,et al.  Deepdocclassifier: Document classification with deep Convolutional Neural Network , 2015, 2015 13th International Conference on Document Analysis and Recognition (ICDAR).

[31]  E. Frauenknecht,et al.  WZIS – Wasserzeichen-Informationssystem: Verwaltung und Präsentation von Wasserzeichen und ihrer Metadaten , 2015 .

[32]  Andrew Zisserman,et al.  The State of the Art: Object Retrieval in Paintings using Discriminative Regions , 2014, BMVC.

[33]  Hongping Cai,et al.  The Cross-Depiction Problem: Computer Vision Algorithms for Recognising Objects in Artwork and in Photographs , 2015, ArXiv.

[34]  Lambert Schomaker,et al.  Image-based historical manuscript dating using contour and stroke fragments , 2016, Pattern Recognit..

[35]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.