Visual Link Retrieval in a Database of Paintings

This paper examines how far state-of-the-art machine vision algorithms can be used to retrieve common visual patterns shared by series of paintings. The research of such visual patterns, central to Art History Research, is challenging because of the diversity of similarity criteria that could relevantly demonstrate genealogical links. We design a methodology and a tool to annotate efficiently clusters of similar paintings and test various algorithms in a retrieval task. We show that pre-trained convolutional neural network can perform better for this task than other machine vision methods aimed at photograph analysis. We also show that retrieval performance can be significantly improved by fine-tuning a network specifically for this task.

[1]  Yang Song,et al.  Learning Fine-Grained Image Similarity with Deep Ranking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  J. M. Hughes,et al.  Quantification of artistic style through sparse coding analysis in the drawings of Pieter Bruegel the Elder , 2010, Proceedings of the National Academy of Sciences.

[3]  Mathieu Aubry,et al.  Painting-to-3D model alignment via discriminative visual elements , 2014, TOGS.

[4]  Thomas Mensink,et al.  The Rijksmuseum Challenge: Museum-Centered Visual Recognition , 2014, ICMR.

[5]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[6]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[7]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[8]  Leonidas J. Guibas,et al.  Shape google: Geometric words and expressions for invariant shape retrieval , 2011, TOGS.

[9]  Bertrand Jestaz.  Krzysztof Pomian, Collectionneurs, amateurs et curieux. Paris, Venise : XVIe-XVIIIe siècle. , 1987 .

[10]  Victor S. Lempitsky,et al.  Neural Codes for Image Retrieval , 2014, ECCV.

[11]  Peter Paul Sir Rubens,et al.  Rubens and His Legacy , 2014 .

[12]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[13]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[14]  Victor S. Lempitsky,et al.  Aggregating Local Deep Features for Image Retrieval , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[15]  Marianne Bradnock.  Google art project , 2011 .

[16]  Michael Isard,et al.  Object retrieval with large vocabularies and fast spatial matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Alexei A. Efros,et al.  Data-driven visual similarity for cross-domain image matching , 2011, ACM Trans. Graph..

[18]  Babak Saleh,et al.  Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature , 2015, ArXiv.

[19]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[20]  Brian Wyvill,et al.  Robust iso-surface tracking for interactive character skinning , 2014, ACM Trans. Graph..

[21]  Atsuto Maki,et al.  A Baseline for Visual Instance Retrieval with Deep Convolutional Networks , 2014, ICLR 2015.

[22]  Alessio Del Bue,et al.  Artistic Image Classification: An Analysis on the PRINTART Database , 2012, ECCV.

[23]  David Picard,et al.  Challenges in Content-Based Image Indexing of Cultural Heritage Collections , 2015, IEEE Signal Processing Magazine.

[24]  Ying Wu,et al.  Object retrieval and localization with spatially-constrained similarity measure and k-NN re-ranking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  Jay Tribby,et al.  Collectionneurs, amateurs et curieux : Paris, Venise, XVIe-XVIIIe siècle , 1988 .

[27]  Jessica M. Hollands.  Web Gallery of Art , 2001 .

[28]  Andrew Zisserman,et al.  In Search of Art , 2014, ECCV Workshops.

[29]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[30]  Yann LeCun,et al.  Dimensionality Reduction by Learning an Invariant Mapping , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[31]  T. Kaufmann,et al.  Toward a Geography of Art , 2004 .

[32]  Babak Saleh,et al.  Quantifying Creativity in Art Networks , 2015, ICCC.

[33]  Atsuto Maki,et al.  From generic to specific deep representations for visual recognition , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[34]  Bernard Aikema,et al.  Le Botteghe di Tiziano , 2010 .

[35]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Lior Wolf,et al.  Classification of Artistic Styles Using Binarized Features Derived from a Deep Neural Network , 2014, ECCV Workshops.

[37]  Andrew Zisserman,et al.  The State of the Art: Object Retrieval in Paintings using Discriminative Regions , 2014, BMVC.

[38]  Cordelia Schmid,et al.  Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search , 2008, ECCV.

[39]  Trevor Darrell,et al.  Recognizing Image Style , 2013, BMVC.