A deep learning approach to clustering visual arts

Clustering artworks is difficult for several reasons. On the one hand, recognizing meaningful patterns based on domain knowledge and visual perception is extremely hard. On the other hand, applying traditional clustering and feature reduction techniques to the highly dimensional pixel space can be ineffective. To address these issues, in this paper we propose DELIUS: a DEep learning approach to cLustering vIsUal artS. The method uses a pre-trained convolutional network to extract features and then feeds these features into a deep embedded clustering model, where the task of mapping the raw input data to a latent space is jointly optimized with the task of finding a set of cluster centroids in this latent space. Quantitative and qualitative experimental results show the effectiveness of the proposed method. DELIUS can be useful for several tasks related to art analysis, in particular visual link retrieval and historical knowledge discovery in painting datasets.

[1]  Marcel Worring,et al.  OmniArt: Multi-task Deep Learning for Artistic Data Analysis , 2017, ArXiv.

[2]  Jufeng Yang,et al.  Recognizing the Style of Visual Arts via Adaptive Cross-layer Correlation , 2019, ACM Multimedia.

[3]  Giovanna Castellano,et al.  Visual link retrieval and knowledge discovery in painting datasets. , 2020 .

[4]  David A. Forsyth,et al.  Clustering art , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[5]  Hongping Cai,et al.  The Cross-Depiction Problem: Computer Vision Algorithms for Recognising Objects in Artwork and in Photographs , 2015, ArXiv.

[6]  Shang Gao,et al.  Deep clustering of protein folding simulations , 2018, BMC Bioinformatics.

[7]  Ali Farhadi,et al.  Unsupervised Deep Embedding for Clustering Analysis , 2015, ICML.

[8]  Kiyoshi Tanaka,et al.  Improved ArtGAN for Conditional Synthesis of Natural Image and Artwork , 2017, IEEE Transactions on Image Processing.

[9]  Rita Cucchiara,et al.  Art2Real: Unfolding the Reality of Artworks via Semantically-Aware Image-To-Image Translation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Giovanna Castellano,et al.  Deep Convolutional Embedding for Digitized Painting Clustering , 2021, 2020 25th International Conference on Pattern Recognition (ICPR).

[11]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[12]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  En Zhu,et al.  Deep Clustering with Convolutional Autoencoders , 2017, ICONIP.

[15]  Lior Shamir,et al.  Impressionism, expressionism, surrealism: Automated recognition of painters and schools of art , 2010, TAP.

[16]  Hongping Cai,et al.  Detecting People in Artwork with CNNs , 2016, ECCV Workshops.

[17]  Trevor Darrell,et al.  Recognizing Image Style , 2013, BMVC.

[18]  Rita Cucchiara,et al.  Explaining digital humanities by aligning images and textual descriptions , 2020, Pattern Recognit. Lett..

[19]  Michael Felsberg,et al.  Painting-91: a large scale database for computational painting categorization , 2014, Machine Vision and Applications.

[20]  Zenglin Xu,et al.  Semi-supervised deep embedded clustering , 2019, Neurocomputing.

[21]  Ahmed M. Elgammal,et al.  CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms , 2017, ICCC.

[22]  Babak Saleh,et al.  Toward automated discovery of artistic influence , 2014, Multimedia Tools and Applications.

[23]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  T. Caliński,et al.  A dendrite method for cluster analysis , 1974 .

[25]  Alexei A. Efros,et al.  Discovering Visual Patterns in Art Collections With Spatially-Consistent Feature Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Sonja Grgic,et al.  A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art , 2019, IEEE Access.

[27]  H. Leder,et al.  A model of aesthetic appreciation and aesthetic judgments. , 2004, British journal of psychology.

[28]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[29]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[30]  Ravneet Singh Arora,et al.  Towards automated classification of fine-art painting style: A comparative study , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[31]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Saïd Ladjal,et al.  Weakly Supervised Object Detection in Artworks , 2018, ECCV Workshops.

[33]  Zhiyao Duan,et al.  Audio–Visual Deep Clustering for Speech Separation , 2019, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[34]  George Vogiatzis,et al.  How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval , 2018, ECCV Workshops.

[35]  Kiyoshi Tanaka,et al.  Ceci n'est pas une pipe: A deep convolutional network for fine-art paintings classification , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[36]  Bo Yang,et al.  Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering , 2016, ICML.

[37]  Jiawei Han,et al.  Locally Consistent Concept Factorization for Document Clustering , 2011, IEEE Transactions on Knowledge and Data Engineering.

[38]  Giovanna Castellano,et al.  Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview , 2021, Neural Computing and Applications.

[39]  Qi Wu,et al.  Beyond Photo-Domain Object Recognition: Benchmarks for the Cross-Depiction Problem , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[40]  Christian Wallraven,et al.  Image Statistics for Clustering Paintings According to their Visual Appearance , 2009, CAe.

[41]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[42]  Alessio Del Bue,et al.  Artistic Image Classification: An Analysis on the PRINTART Database , 2012, ECCV.

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Andrew Zisserman,et al.  In Search of Art , 2014, ECCV Workshops.

[45]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[46]  Shang Gao,et al.  Deep clustering of protein folding simulations , 2018 .

[47]  Tomislav Lipic,et al.  Fine-tuning Convolutional Neural Networks for fine art classification , 2018, Expert Syst. Appl..

[48]  Masoud Makrehchi,et al.  Predicting and Grouping Digitized Paintings by Style using Unsupervised Feature Learning. , 2017, Journal of cultural heritage.

[49]  Yuta Nakashima,et al.  ContextNet: representation and exploration for painting classification and retrieval in context , 2019, International Journal of Multimedia Information Retrieval.

[50]  Eric O. Postma,et al.  Toward Discovery of the Artist's Style: Learning to recognize artists by their artworks , 2015, IEEE Signal Processing Magazine.

[51]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.