A Novel Feature Fusion Method for Computing Image Aesthetic Quality

Computationally, the aesthetic quality of an image means that the model automatically scores the aesthetic level of the image. However, there are many factors that determine beauty or ugliness for photographic photos. Therefore, extracting a variety of representative aesthetic features and fusing these features are still difficult tasks. In this paper, we design a two-stream network to calculate the aesthetic quality of the image. The upper stream of the network is an improved network with the SEResNet-50 and six skip connections added, which can improve the performance of the model without training to obtain deep convolutional neural network features. The lower stream of the network consists of the proposed algorithms for handcrafted extracting aesthetic features and multiple convolution layers to extract the aesthetic features. Finally, to fuse the features of the two-stream network without adding feature dimensions, a novel feature fusion layer is proposed. The results show that this novel feature fusion method can calculate results close to the artificial aesthetic evaluation.

[1]  Sebastian Bosse,et al.  Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment , 2016, IEEE Transactions on Image Processing.

[2]  Florent Perronnin,et al.  Learning beautiful (and ugly) attributes , 2013, BMVC.

[3]  Rangachar Kasturi,et al.  Machine vision , 1995 .

[4]  Yimin Zhou,et al.  Photograph aesthetical evaluation and classification with deep convolutional neural networks , 2017, Neurocomputing.

[5]  Xiaoou Tang,et al.  Image Aesthetic Assessment: An experimental survey , 2016, IEEE Signal Processing Magazine.

[6]  James Ze Wang,et al.  Studying Aesthetics in Photographic Images Using a Computational Approach , 2006, ECCV.

[7]  Rita Cucchiara,et al.  Scene-driven Retrieval in Edited Videos using Aesthetic and Semantic Deep Features , 2016, ICMR.

[8]  Houqiang Li,et al.  Photo Quality Assessment with DCNN that Understands Image Well , 2015, MMM.

[9]  Xiaogang Wang,et al.  Content-based photo quality assessment , 2011, 2011 International Conference on Computer Vision.

[10]  James Zijun Wang,et al.  RAPID: Rating Pictorial Aesthetics using Deep Learning , 2014, ACM Multimedia.

[11]  Jar-Ferr Yang,et al.  Linear Discriminant Regression Classification for Face Recognition , 2013, IEEE Signal Processing Letters.

[12]  Gabriela Csurka,et al.  Assessing the aesthetic quality of photographs using generic image descriptors , 2011, 2011 International Conference on Computer Vision.

[13]  Henry Kang,et al.  Photo Aesthetics Analysis via DCNN Feature Encoding , 2017, IEEE Transactions on Multimedia.

[14]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Shuang Ma,et al.  A-Lamp: Adaptive Layout-Aware Multi-patch Deep Convolutional Neural Network for Photo Aesthetic Assessment , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Lawrence G. Roberts,et al.  Machine Perception of Three-Dimensional Solids , 1963, Outstanding Dissertations in the Computer Sciences.

[17]  Haibin Ling,et al.  A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Ran He,et al.  Deep Aesthetic Quality Assessment With Semantic Information , 2016, IEEE Transactions on Image Processing.

[19]  Radomír Mech,et al.  Photo Aesthetics Ranking Network with Attributes and Content Adaptation , 2016, ECCV.

[20]  Julie Adair King,et al.  Digital Photography Handbook , 2001 .

[21]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Tao Mei,et al.  Query-Dependent Aesthetic Model With Deep Learning for Photo Quality Assessment , 2015, IEEE Transactions on Multimedia.

[23]  Yan Ke,et al.  The Design of High-Level Features for Photo Quality Assessment , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[24]  W. Chu.  Studying Aesthetics in Photographic Images Using a Computational Approach , 2013 .

[25]  Radomír Mech,et al.  Deep Multi-patch Aggregation Network for Image Style, Aesthetics, and Quality Estimation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[26]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[27]  Ishu Arora,et al.  Bi-featured image quality assessment with the hierarchical image quality enhancement algorithm , 2016, 2016 International Conference on Inventive Computation Technologies (ICICT).

[28]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Naila Murray,et al.  AVA: A large-scale database for aesthetic visual analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Le Wu,et al.  ILGNet: inception modules with connected local and global features for efficient image aesthetic quality classification using domain adaptation , 2016, IET Comput. Vis..

[31]  Sajid Saleem,et al.  A Robust SIFT Descriptor for Multispectral Images , 2014, IEEE Signal Processing Letters.

[32]  John See,et al.  Filling the gaps: Reducing the complexity of networks for multi-attribute image aesthetic prediction , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[33]  Vicente Ordonez,et al.  High level describable attributes for predicting aesthetics and interestingness , 2011, CVPR 2011.

[34]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Daniel Cohen-Or,et al.  Color harmonization , 2006, ACM Trans. Graph..

[36]  Damon M. Chandler,et al.  Opinion-Unaware Blind Quality Assessment of Multiply and Singly Distorted Images via Distortion Parameter Estimation , 2018, IEEE Transactions on Image Processing.

[37]  Xin Fu,et al.  Image Aesthetics Assessment Using Composite Features from off-the-Shelf Deep Models , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[38]  Xin Wang,et al.  Laplacian Operator-Based Edge Detectors , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Xinbo Gao,et al.  A Gated Peripheral-Foveal Convolutional Neural Network for Unified Image Aesthetic Prediction , 2018, IEEE Transactions on Multimedia.

[40]  Hendrik P. A. Lensch,et al.  Will People Like Your Image? Learning the Aesthetic Space , 2016, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[41]  David W. Messinger,et al.  Leveraging Expert Feature Knowledge for Predicting Image Aesthetics , 2018, IEEE Transactions on Image Processing.