Artist-based painting classification using Markov random fields with convolution neural network

Determining the authorship of a painting image is a challenging task because paintings of an artist may not have a unique style and various artists may have similar painting styles. In this paper, we present a new approach to categorize digital painting images based on artist. We construct a multi-scale pyramid from a painting image to consider both globally and locally the information contained in one image. For each layer, we train a Convolutional Neural Network (CNN) model to determine the class label. To build the relationship within local image patches, we employ Markov random fields (MRFs) by optimizing the Gibbs energy function defined by (1) the data term measuring the compatibility of labeling with given data, and (2) the smoothness term penalizing assignments that label neighboring patches differently. A new fusion scheme is proposed to aggregate patch-level classification results. The proposed CNN-MRF method is validated using two challenging painting image datasets. Experimental results show that the proposed method is effective and achieves state-of-the-art performance.

[1]  Luiz Eduardo Soares de Oliveira,et al.  Multiple instance learning for histopathological breast cancer image classification , 2019, Expert Syst. Appl..

[2]  Xili Wang,et al.  A novel grouped sparse representation for face recognition , 2018, Multimedia Tools and Applications.

[3]  Simon Perathoner,et al.  Deep learning image recognition enables efficient genome editing in zebrafish by automated injections , 2018, bioRxiv.

[4]  Kathiravan Srinivasan,et al.  Robust RGB-D Hand Tracking Using Deep Learning Priors , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[5]  Yu Liu,et al.  CNN-RNN: a large-scale hierarchical image classification framework , 2018, Multimedia Tools and Applications.

[6]  Beng Chin Ooi,et al.  Effective Multi-Modal Retrieval based on Stacked Auto-Encoders , 2014, Proc. VLDB Endow..

[7]  Shuai Wang,et al.  Deep learning for sentiment analysis: A survey , 2018, WIREs Data Mining Knowl. Discov..

[8]  Hanchao Qi,et al.  A new method for visual stylometry on impressionist paintings , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Tsuhan Chen,et al.  Cross-layer features in convolutional neural networks for generic classification tasks , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[10]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[11]  Ruifan Li,et al.  Cross-modal Retrieval with Correspondence Autoencoder , 2014, ACM Multimedia.

[12]  Y. Liu,et al.  Bilinear deep learning for image classification , 2011, ACM Multimedia.

[13]  William T. Freeman,et al.  Learning low-level vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[14]  Wen-Huang Cheng,et al.  Computer-aided classification of lung nodules on computed tomography images via deep learning technique , 2015, OncoTargets and therapy.

[15]  Duanqing Xu,et al.  Optimal Transport of Deep Feature for Image Style Transfer , 2019, ICMSSP 2019.

[16]  Margaret Lech,et al.  Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings , 2019, IEEE Access.

[17]  Minh N. Do,et al.  A revisit to MRF-based depth map super-resolution and enhancement , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[19]  Beng Chin Ooi,et al.  Deep Learning at Scale and at Ease , 2016, ACM Trans. Multim. Comput. Commun. Appl..

[20]  Kien A. Hua,et al.  Field Effect Deep Networks for Image Recognition with Incomplete Data , 2016, ACM Trans. Multim. Comput. Commun. Appl..

[21]  Raymond Y. K. Lau,et al.  Hyperspectral Image Classification With Deep Learning Models , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Kai-Lung Hua,et al.  Fashion World Map: Understanding Cities Through Streetwear Fashion , 2017, ACM Multimedia.

[23]  Mei-Chen Yeh,et al.  Artist-based Classification via Deep Learning with Multi-scale Weighted Pooling , 2016, ACM Multimedia.

[24]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[25]  Tulay Yildirim,et al.  Painter Classification Over the Novel Art Painting Data Set via The Latest Deep Neural Networks , 2019, Procedia Computer Science.

[26]  Kai-Lung Hua,et al.  Depth map super-resolution via Markov Random Fields without texture-copying artifacts , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[27]  Yue Qi,et al.  A Novel Framework of Hand Localization and Hand Pose Estimation , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  P. Pérez,et al.  Markov random fields and images , 1998 .

[30]  Hongxun Yao,et al.  Predicting discrete probability distribution of image emotions , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[31]  Tarpit Sahu,et al.  Indian Art Form Recognition Using Convolutional Neural Networks , 2018, 2018 5th International Conference on Signal Processing and Integrated Networks (SPIN).

[32]  Kiyoshi Tanaka,et al.  Ceci n'est pas une pipe: A deep convolutional network for fine-art paintings classification , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[33]  Henry Leung,et al.  Outdoor Semantic Segmentation for UGVs Based on CNN and Fully Connected CRFs , 2019, IEEE Sensors Journal.

[34]  Sheng-hua Zhong,et al.  Fine-art painting classification via two-channel dual path networks , 2019, Int. J. Mach. Learn. Cybern..

[35]  Jinchang Ren,et al.  Brushstroke based sparse hybrid convolutional neural networks for author classification of Chinese ink-wash paintings , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[36]  Peng Zhang,et al.  Affective video content analysis based on multimodal data fusion in heterogeneous networks , 2019, Inf. Fusion.

[37]  Deep Medhi,et al.  A study on layer correlation effects through a multilayer network optimization problem , 2011, 2011 23rd International Teletraffic Congress (ITC).

[38]  J. M. Hammersley,et al.  Markov fields on finite graphs and lattices , 1971 .

[39]  Daeyoung Kim,et al.  High-quality depth map up-sampling robust to edge noise of range sensors , 2012, 2012 19th IEEE International Conference on Image Processing.

[40]  Jin Young Lee,et al.  Deep learning ensemble with data augmentation using a transcoder in visual description , 2019, Multimedia Tools and Applications.

[41]  Aleš Leonardis,et al.  Exploring Object-Centric and Scene-Centric CNN Features and Their Complementarity for Human Rights Violations Recognition in Images , 2018, IEEE Access.

[42]  Mei-Chen Yeh,et al.  A CNN-LSTM framework for authorship classification of paintings , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[43]  Joel J. P. C. Rodrigues,et al.  A novel deep learning based framework for the detection and classification of breast cancer using transfer learning , 2019, Pattern Recognit. Lett..

[44]  Xiaoxiao Li,et al.  Deep Learning Markov Random Field for Semantic Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.