OmniArt: Multi-task Deep Learning for Artistic Data Analysis

Vast amounts of artistic data is scattered on-line from both museums and art applications. Collecting, processing and studying it with respect to all accompanying attributes is an expensive process. With a motivation to speed up and improve the quality of categorical analysis in the artistic domain, in this paper we propose an efficient and accurate method for multi-task learning with a shared representation applied in the artistic domain. We continue to show how different multi-task configurations of our method behave on artistic data and outperform handcrafted feature approaches as well as convolutional neural networks. In addition to the method and analysis, we propose a challenge like nature to the new aggregated data set with almost half a million samples and structured meta-data to encourage further research and societal engagement.

[1]  Cees Snoek,et al.  Academia Meets Industry at the Multimedia Grand Challenge , 2011, IEEE Multim..

[2]  J. C. Rush.  Acquiring a Concept of Painting Style , 1979 .

[3]  D. Sculley,et al.  Combined regression and ranking , 2010, KDD.

[4]  Jean-Marc Odobez,et al.  Transferring Neural Representations for Low-Dimensional Indexing of Maya Hieroglyphic Art , 2016, ECCV Workshops.

[5]  Jitendra Malik,et al.  Detecting People in Cubist Art , 2014, ECCV Workshops.

[6]  Xiaodong Liu,et al.  Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval , 2015, NAACL.

[7]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[8]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  T. Kaufmann,et al.  Toward a Geography of Art , 2004 .

[10]  Yongxin Yang,et al.  A Unified Perspective on Multi-Domain and Multi-Task Learning , 2014, ICLR.

[11]  Xun Xu,et al.  Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation , 2016, ECCV.

[12]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[13]  Koen Janssens,et al.  2D X-ray and FTIR micro-analysis of the degradation of cadmium yellow pigment in paintings of Henri Matisse , 2015 .

[14]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[15]  Babak Saleh,et al.  Quantifying Creativity in Art Networks , 2015, ICCC.

[16]  Lior Wolf,et al.  Classification of Artistic Styles Using Binarized Features Derived from a Deep Neural Network , 2014, ECCV Workshops.

[17]  Luc Moens,et al.  Analysis of pigments and inks on oil paintings and historical manuscripts using total reflection x-ray fluorescence spectrometry , 2000 .

[18]  Marcel van Gerven,et al.  Convolutional Sketch Inversion , 2016, ECCV Workshops.

[19]  Hongping Cai,et al.  Detecting People in Artwork with CNNs , 2016, ECCV Workshops.

[20]  Trevor Darrell,et al.  Recognizing Image Style , 2013, BMVC.

[21]  Thomas Mensink,et al.  The Rijksmuseum Challenge: Museum-Centered Visual Recognition , 2014, ICMR.

[22]  Iasonas Kokkinos,et al.  UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Eric O. Postma,et al.  Toward Discovery of the Artist's Style: Learning to recognize artists by their artworks , 2015, IEEE Signal Processing Magazine.

[24]  E. LESTER SMITH,et al.  AND OTHERS , 2005 .

[25]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Kunio Kashino,et al.  Adaptive Visual Feedback Generation for Facial Expression Improvement with Multi-task Deep Neural Networks , 2016, ACM Multimedia.

[27]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[28]  F. Chung,et al.  Industrial Applications of X-Ray Diffraction , 1999 .

[29]  Jia Li,et al.  Image processing for artist identification , 2008, IEEE Signal Processing Magazine.

[30]  Andrew Zisserman,et al.  In Search of Art , 2014, ECCV Workshops.

[31]  Babak Saleh,et al.  Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature , 2015, ArXiv.

[32]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Wayne Franits,et al.  Dutch Seventeenth-Century Genre Painting: Its Stylistic and Thematic Evolution , 2004 .

[34]  Hui Cheng,et al.  Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning , 2016, ACM Multimedia.

[35]  Huibert Crijns,et al.  Memory of the Netherlands: Past and Future of the Dutch National Database on Cultural Heritage , 2009 .

[36]  Martial Hebert,et al.  Cross-Stitch Networks for Multi-task Learning , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[38]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[39]  Koen E. A. van de Sande,et al.  Evaluating Color Descriptors for Object and Scene Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[41]  L. D. Couprie.  Iconclass: an iconographic classification system , 1983 .

[42]  Lora Aroyo,et al.  Semantic annotation and search of cultural-heritage collections: The MultimediaN E-Culture demonstrator , 2008, J. Web Semant..

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Shawn D. Newsam,et al.  Efficient Action Detection in Untrimmed Videos via Multi-task Learning , 2016, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[45]  Lexing Ying,et al.  Quantitative Canvas Weave Analysis Using 2-D Synchrosqueezed Transforms: Application of time-frequency analysis to art investigation , 2015, IEEE Signal Processing Magazine.