Representation learning of image composition for aesthetic prediction

Abstract Photo quality assessment (PQA) aims at computationally and precisely evaluating the quality of images from the aspect of aesthetic. Image aesthetic is strongly correlated with composition. However, few existing works have taken composition into consideration. Besides, existing composition features are typically hand-crafted. In this paper, we propose a novel end-to-end framework for representation learning of image composition. Specially, we build a fully connected graph based on deep features in Convolutional Neural Networks (CNNs). In the graph, edge attributes i.e. similarities between deep features at different positions are used for representing image composition. Besides, we use global attributes of the graph to represent miscellaneous aesthetic aspects. Finally, we use a gate unit to combine both composition features and miscellaneous aesthetic features for aesthetic prediction. The whole network can be trained in an end-to-end manner. Experimental results show that the proposed techniques significantly improves the prediction precision of aesthetic and composition over various datasets. We have released our codes at: .

[1]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[2]  Fei Gao,et al.  DeepSim: Deep similarity for image quality assessment , 2017, Neurocomputing.

[3]  Jun Yu,et al.  Multitask Autoencoder Model for Recovering Human Poses , 2018, IEEE Transactions on Industrial Electronics.

[4]  Andreas E. Savakis,et al.  Evaluation of image appeal in consumer photography , 2000, Electronic Imaging.

[5]  Xinbo Gao,et al.  A Gated Peripheral-Foveal Convolutional Neural Network for Unified Image Aesthetic Prediction , 2018, IEEE Transactions on Multimedia.

[6]  Qi Tian,et al.  Blind image quality prediction by exploiting multi-level deep representations , 2018, Pattern Recognit..

[7]  Sonja Grgic,et al.  A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art , 2019, IEEE Access.

[8]  Radomír Mech,et al.  Photo Aesthetics Ranking Network with Attributes and Content Adaptation , 2016, ECCV.

[9]  Junchi Yan,et al.  Collaborative and Attentive Learning for Personalized Image Aesthetic Assessment , 2018, IJCAI.

[10]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Joachim Denzler,et al.  JenAesthetics Subjective Dataset: Analyzing Paintings by Subjective Scores , 2014, ECCV Workshops.

[12]  Peyman Milanfar,et al.  NIMA: Neural Image Assessment , 2017, IEEE Transactions on Image Processing.

[13]  Xiaoou Tang,et al.  Image Aesthetic Assessment: An experimental survey , 2016, IEEE Signal Processing Magazine.

[14]  Luming Zhang,et al.  Engineering Deep Representations for Modeling Aesthetic Perception , 2018, IEEE Transactions on Cybernetics.

[15]  Le Wu,et al.  Predicting Aesthetic Score Distribution through Cumulative Jensen-Shannon Divergence , 2017, AAAI.

[16]  Houqiang Li,et al.  Photo Quality Assessment with DCNN that Understands Image Well , 2015, MMM.

[17]  Ran He,et al.  Visual Aesthetic Quality Assessment with Multi-task Deep Learning , 2016, ArXiv.

[18]  Fei Gao,et al.  MLANs: Image Aesthetic Assessment via Multi-Layer Aggregation Networks , 2018, 2018 Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA).

[19]  Thomas S. Huang,et al.  Brain-Inspired Deep Networks for Image Aesthetics Assessment , 2016, ArXiv.

[20]  Ran He,et al.  Deep Aesthetic Quality Assessment With Semantic Information , 2016, IEEE Transactions on Image Processing.

[21]  Jun Yu,et al.  Hierarchical Deep Click Feature Prediction for Fine-Grained Image Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Stephen Lin,et al.  Change-Based Image Cropping with Exclusion and Compositional Features , 2015, International Journal of Computer Vision.

[23]  Chang-Su Kim,et al.  Property-Specific Aesthetic Assessment With Unsupervised Aesthetic Property Discovery , 2019, IEEE Access.

[24]  Le Wu,et al.  ILGNet: inception modules with connected local and global features for efficient image aesthetic quality classification using domain adaptation , 2016, IET Comput. Vis..

[25]  Qingming Huang,et al.  Style-adaptive photo aesthetic rating via convolutional neural networks and multi-task learning , 2020, Neurocomputing.

[26]  In-So Kweon,et al.  Modeling photo composition and its application to photo re-arrangement , 2012, 2012 19th IEEE International Conference on Image Processing.

[27]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[28]  Vasileios Mezaris,et al.  Image Aesthetics Assessment Using Fully Convolutional Neural Networks , 2019, MMM.

[29]  Henry Kang,et al.  Photo Aesthetics Analysis via DCNN Feature Encoding , 2017, IEEE Transactions on Multimedia.

[30]  Fei Gao,et al.  Deep Multimodal Distance Metric Learning Using Click Constraints for Image Ranking , 2017, IEEE Transactions on Cybernetics.

[31]  Fei Gao,et al.  Biologically inspired image quality assessment , 2016, Signal Process..

[32]  Jianping Fan,et al.  iPrivacy: Image Privacy Protection by Identifying Sensitive Objects via Deep Multi-Task Learning , 2017, IEEE Transactions on Information Forensics and Security.

[33]  Radomír Mech,et al.  Deep Multi-patch Aggregation Network for Image Style, Aesthetics, and Quality Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[34]  Kwan-Liu Ma,et al.  Learning to Compose with Professional Photographs on the Web , 2017, ACM Multimedia.

[35]  Shuang Ma,et al.  A-Lamp: Adaptive Layout-Aware Multi-patch Deep Convolutional Neural Network for Photo Aesthetic Assessment , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[37]  Jie Li,et al.  Beyond Vision: A Multimodal Recurrent Attention Convolutional Neural Network for Unified Image Aesthetic Prediction Tasks , 2021, IEEE Transactions on Multimedia.

[38]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.