Integrating Contextual Knowledge to Visual Features for Fine Art Classification

Automatic art analysis has seen an ever-increasing interest from the pattern recognition and computer vision community. However, most of the current work is mainly based solely on digitized artwork images, sometimes supplemented with some metadata and textual comments. A knowledge graph that integrates a rich body of information about artworks, artists, painting schools, etc., in a unified structured framework can provide a valuable resource for more powerful information retrieval and knowledge discovery tools in the artistic domain. To this end, this paper presents ArtGraph: an artistic knowledge graph based on WikiArt and DBpedia. The graph, implemented in Neo4j, already provides knowledge discovery capabilities without having to train a learning system. In addition, the embeddings extracted from the graph are used to inject “contextual” knowledge into a deep learning model to improve the accuracy of artwork attribute prediction tasks.

[1]  George Vogiatzis,et al.  How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval , 2018, ECCV Workshops.

[2]  Hajime Nagahara,et al.  GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph , 2021, ICMR.

[3]  Margaret Lech,et al.  Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings , 2019, IEEE Access.

[4]  Thomas de Quincey.  [C] , 2000, The Works of Thomas De Quincey, Vol. 1: Writings, 1799–1820.

[5]  Tomislav Lipic,et al.  Fine-tuning Convolutional Neural Networks for fine art classification , 2018, Expert Syst. Appl..

[6]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[7]  Palash Goyal,et al.  Graph Embedding Techniques, Applications, and Performance: A Survey , 2017, Knowl. Based Syst..

[8]  Giovanna Castellano,et al.  Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview , 2021, Neural Computing and Applications.

[9]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[10]  Yuta Nakashima,et al.  ContextNet: representation and exploration for painting classification and retrieval in context , 2019, International Journal of Multimedia Information Retrieval.

[11]  Steffen Staab,et al.  Knowledge graphs , 2021, Commun. ACM.

[12]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Giacomo Mercuriali.  Digital Art History and the Computational Imagination , 2018 .