Automatic Matching of Paintings and Descriptions in Art-Historic Archives using Multimodal Analysis

Cultural heritage data plays a pivotal role in the understanding of human history and culture. A wealth of information is buried in art-historic archives which can be extracted via digitization and analysis. This information can facilitate search and browsing, help art historians to track the provenance of artworks and enable wider semantic text exploration for digital cultural resources. However, this information is contained in images of artworks, as well as textual descriptions or annotations accompanied with the images. During the digitization of such resources, the valuable associations between the images and texts are frequently lost. In this project description, we propose an approach to retrieve the associations between images and texts for artworks from art-historic archives. To this end, we use machine learning to generate text descriptions for the extracted images on the one hand, and to detect descriptive phrases and titles of images from the text on the other hand. Finally, we use embeddings to align both, the descriptions and the images.

[1]  Antoine Isaac,et al.  Supporting Linked Data Production for Cultural Heritage Institutes: The Amsterdam Museum Case Study , 2012, ESWC.

[2]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[3]  Kimmo Kettunen,et al.  Names, Right or Wrong: Named Entities in an OCRed Historical Finnish Newspaper Collection , 2017, DATeCH.

[4]  Fei Liu,et al.  MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance , 2019, EMNLP.

[5]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[6]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[7]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[8]  Tobias Blanke,et al.  Comparison of named entity recognition tools for raw OCR text , 2012, KONVENS.

[9]  Rada Mihalcea,et al.  TextRank: Bringing Order into Text , 2004, EMNLP.

[10]  Elizabeth Alvey.  Linked data for libraries, archives and museums: how to clean, link and publish your metadata , 2016 .

[11]  Ralf Krestel,et al.  Who is Mona L.? Identifying Mentions of Artworks in Historical Archives , 2019, TPDL.

[12]  Lora Aroyo,et al.  The Rijksmuseum collection as Linked Data , 2018, Semantic Web.

[13]  Bum Mook Oh,et al.  Classifying digitized art type and time period , 2018 .

[14]  Rik Van de Walle,et al.  Exploring entity recognition and disambiguation for cultural heritage collections , 2015, Digit. Scholarsh. Humanit..

[15]  Basura Fernando,et al.  SPICE: Semantic Propositional Image Caption Evaluation , 2016, ECCV.

[16]  Erwin M. Bakker,et al.  CycleMatch: A cycle-consistent embedding network for image-text matching , 2019, Pattern Recognit..

[17]  Madely du Preez.  Linked Data for Libraries, Archives and Museums: How to Clean, Link and Publish your Metadata , 2015, Electron. Libr..

[18]  Frédéric Kaplan,et al.  Diachronic Evaluation of NER Systems on Old Newspapers , 2016, KONVENS.

[19]  Anette Hulth,et al.  Automatic Keyword Extraction Using Domain Knowledge , 2001, CICLing.

[20]  Xin Jiang,et al.  A ranking approach to keyphrase extraction , 2009, SIGIR.

[21]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Mark Levene,et al.  Finding Parallel Passages in Cultural Heritage Archives , 2018, ACM Journal on Computing and Cultural Heritage.

[23]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[24]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments , 2005, IEEvaluation@ACL.

[25]  Sheng-hua Zhong,et al.  Fine-Art Painting Classification via Two-Channel Deep Residual Network , 2017, PCM.

[26]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[27]  Jun Wang,et al.  Attention-Aware Multi-Stroke Style Transfer , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Mohamed Elhoseiny,et al.  The Shape of Art History in the Eyes of the Machine , 2018, AAAI.

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  Vincent Ng,et al.  Conundrums in Unsupervised Keyphrase Extraction: Making Sense of the State-of-the-Art , 2010, COLING.

[31]  Florian Yger,et al.  Recognizing Art Style Automatically in Painting with Deep Learning , 2017, ACML.

[32]  Lora Aroyo,et al.  Hacking history via event extraction , 2011, K-CAP '11.

[33]  Kilian Q. Weinberger,et al.  BERTScore: Evaluating Text Generation with BERT , 2019, ICLR.

[34]  Adriana Kovashka,et al.  Artistic Object Recognition by Unsupervised Style Adaptation , 2018, ACCV.

[35]  Ruslan Salakhutdinov,et al.  Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models , 2014, ArXiv.

[36]  Eero Hyvönen,et al.  Knowledge-based Relation Discovery in Cultural Heritage Knowledge Graphs , 2019, DHN.

[37]  C. Lawrence Zitnick,et al.  CIDEr: Consensus-based image description evaluation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Yi-fang Brook Wu,et al.  Domain-specific keyphrase extraction , 2005, CIKM '05.