Classification of basic artistic media based on a deep convolutional approach

Artistic media play an important role in recognizing and classifying artworks in many artwork classification works and public artwork databases. We employ deep CNN structure to recognize artistic media from artworks and to classify them into predetermined categories. For this purpose, we define basic artistic media as oilpaint brush, pastel, pencil and watercolor and build artwork image dataset by collecting artwork images from various websites. To build our classifier, we implement various recent deep CNN structures and compare their performances. Among them, we select DenseNet, which shows best performance for recognizing artistic media. Through the human baseline experiment, we show that the performance of our classifier is compatible with that of trained human. Furthermore, we also show that our classifier shows a similar recognition and classification pattern with human in terms of well-classifying media, ill-classifying media, confusing pair and confusing case. We also collect synthesized oilpaint artwork images from fourteen important oilpaint literatures and apply them to our classifier. Our classifier shows a meaningful performance, which will lead to an evaluation scheme for the artistic media simulation techniques of non-photorealistic rendering (NPR) society.

[1]  Lior Shamir,et al.  Impressionism, expressionism, surrealism: Automated recognition of painters and schools of art , 2010, TAP.

[2]  Frédéric Kaplan,et al.  Visual Link Retrieval in a Database of Paintings , 2016, ECCV Workshops.

[3]  Lior Wolf,et al.  Classification of Artistic Styles Using Binarized Features Derived from a Deep Neural Network , 2014, ECCV Workshops.

[4]  Eli Shechtman,et al.  Example-based synthesis of stylized facial animations , 2017, ACM Trans. Graph..

[5]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Andrew W. Fitzgibbon,et al.  PiCoDes: Learning a Compact Code for Novel-Category Recognition , 2011, NIPS.

[7]  Tobias Isenberg,et al.  State of the "Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video , 2013, IEEE Transactions on Visualization and Computer Graphics.

[8]  Siwei Lyu,et al.  A digital technique for art authentication , 2004, Proc. Natl. Acad. Sci. USA.

[9]  Aaron Hertzmann,et al.  AniPaint: Interactive Painterly Animation from Video , 2012, IEEE Transactions on Visualization and Computer Graphics.

[10]  James She,et al.  DeepArt: Learning Joint Representations of Visual Arts , 2017, ACM Multimedia.

[11]  James Ze Wang,et al.  Studying digital imagery of ancient paintings by mixtures of stochastic models , 2004, IEEE Transactions on Image Processing.

[12]  Stefan Winkler,et al.  Inferring Painting Style with Multi-Task Dictionary Learning , 2015, IJCAI.

[13]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[14]  S. Todorovic,et al.  Video Painting with Space-Time-Varying Style Parameters , 2011, IEEE Transactions on Visualization and Computer Graphics.

[15]  Linda Doyle,et al.  Painting style transfer for head portraits using convolutional neural networks , 2016, ACM Trans. Graph..

[16]  Marcel Worring,et al.  OmniArt: Multi-task Deep Learning for Artistic Data Analysis , 2017, ArXiv.

[17]  Liang Lin,et al.  Painterly animation using video semantics and feature correspondence , 2010, NPAR.

[18]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Luis González Abril,et al.  Categorizing paintings in art styles based on qualitative color descriptors, quantitative global features and machine learning (QArt-Learn) , 2018, Expert Syst. Appl..

[20]  Jian Yang,et al.  Convolution Neural Networks With Two Pathways for Image Style Recognition , 2017, IEEE Transactions on Image Processing.

[21]  Song-Chun Zhu,et al.  Portrait painting using active templates , 2011, NPAR '11.

[22]  Huamin Wang,et al.  Wetbrush: GPU-based 3D painting simulation at the bristle level , 2015, ACM Trans. Graph..

[23]  Peter Litwinowicz,et al.  Processing images and video for an impressionist effect , 1997, SIGGRAPH.

[24]  Fabian Gieseke,et al.  Artistic Movement Recognition by Boosted Fusion of Color Structure and Topographic Description , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[25]  Yu-Ting Tsai,et al.  Generating Pointillism Paintings Based on Seurat's Color Composition , 2013, Comput. Graph. Forum.

[26]  Kiyoshi Tanaka,et al.  Ceci n'est pas une pipe: A deep convolutional network for fine-art paintings classification , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[27]  Irfan A. Essa,et al.  Image and video based painterly animation , 2004, NPAR '04.

[28]  Song-Chun Zhu,et al.  From image parsing to painterly rendering , 2009, TOGS.

[29]  Eric O. Postma,et al.  Toward Discovery of the Artist's Style: Learning to recognize artists by their artworks , 2015, IEEE Signal Processing Magazine.

[30]  David Picard,et al.  Challenges in Content-Based Image Indexing of Cultural Heritage Collections , 2015, IEEE Signal Processing Magazine.

[31]  Tsuhan Chen,et al.  A framework of extracting multi-scale features using multiple convolutional neural networks , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[32]  Fahad Shahbaz Khan,et al.  Combining Holistic and Part-based Deep Representations for Computational Painting Categorization , 2016, ICMR.

[33]  Sonja Grgic,et al.  Genre classification of paintings , 2016, 2016 International Symposium ELMAR.

[34]  Trevor Darrell,et al.  Recognizing Image Style , 2013, BMVC.

[35]  Thomas Mensink,et al.  The Rijksmuseum Challenge: Museum-Centered Visual Recognition , 2014, ICMR.

[36]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[37]  Song-Chun Zhu,et al.  Sisley the abstract painter , 2010, NPAR.

[38]  Daniel Keren,et al.  Painter identification using local features and naive Bayes , 2002, Object recognition supported by user interaction for service robots.

[39]  Aaron Hertzmann,et al.  Painterly rendering with curved brush strokes of multiple sizes , 1998, SIGGRAPH.

[40]  Tomislav Lipic,et al.  Fine-tuning Convolutional Neural Networks for fine art classification , 2018, Expert Syst. Appl..

[41]  Christoph Trattner,et al.  Content-based artwork recommendation: integrating painting metadata with neural and manually-engineered visual features , 2018, User Modeling and User-Adapted Interaction.