Colornet - Estimating Colorfulness in Natural Images

Measuring the colorfulness of a natural or virtual scene is critical for many applications in image processing field ranging from capturing to display. In this paper, we propose the first deep learning-based colorfulness estimation metric. For this purpose, we develop a color rating model which simultaneously learns to extracts the pertinent characteristic color features and the mapping from feature space to the ideal colorfulness scores for a variety of natural colored images. Additionally, we propose to overcome the lack of adequate annotated dataset problem by combining/aligning two publicly available colorfulness databases using the results of a new subjective test which employs a common subset of both databases. Using the obtained subjectively annotated dataset with 180 colored images, we finally demonstrate the efficacy of our proposed model over the traditional methods, both quantitatively and qualitatively.

[1]  Aljoscha Smolic,et al.  Automated Aesthetic Analysis of Photographic Images , 2015, IEEE Transactions on Visualization and Computer Graphics.

[2]  Cagri Ozcinar,et al.  Towards Generating Ambisonics Using Audio-visual Cue for Virtual Reality , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Marcus Barkowsky,et al.  Aligning subjective tests using a low cost common set , 2011 .

[4]  Takanori Hayashi,et al.  Performance comparison of subjective assessment methods for 3D video quality , 2012, 2012 Fourth International Workshop on Quality of Multimedia Experience.

[5]  Giuseppe Valenzise,et al.  Learning-Based Tone Mapping Operator for Efficient Image Matching , 2019, IEEE Transactions on Multimedia.

[6]  W. Chu Studying Aesthetics in Photographic Images Using a Computational Approach , 2013 .

[7]  Marcus Barkowsky,et al.  Selecting scenes for 2D and 3D subjective video quality tests , 2013, EURASIP J. Image Video Process..

[8]  Peter G. Engeldrum,et al.  Psychometric Scaling: A Toolkit for Imaging Systems Development , 2000 .

[9]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[10]  Giuseppe Valenzise,et al.  Learning-based adaptive tone mapping for keypoint detection , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[11]  Erik Reinhard,et al.  A Gamut-Mapping Framework for Color-Accurate Reproduction of HDR Images , 2016, IEEE Computer Graphics and Applications.

[12]  Yoshinobu Nayatani,et al.  Revision of the chroma and hue scales of a nonlinear color‐appearance model , 1995 .

[13]  Wolfgang Heidrich,et al.  Color correction for tone mapping , 2009, Comput. Graph. Forum.

[14]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[15]  Michael H. Brill,et al.  Color appearance models , 1998 .

[16]  Karel Fliegel,et al.  Objective evaluation of naturalness, contrast, and colorfulness of tone-mapped images , 2014, Optics & Photonics - Optical Engineering + Applications.

[17]  Michael Wimmer,et al.  Evaluation of HDR tone mapping methods using essential perceptual attributes , 2008, Comput. Graph..

[18]  Huib de Ridder,et al.  Optimizing color reproduction of natural images , 1998, CIC.

[19]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Sos S. Agaian,et al.  No reference color image contrast and quality measures , 2013, IEEE Transactions on Consumer Electronics.

[21]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[22]  Peyman Milanfar,et al.  NIMA: Neural Image Assessment , 2017, IEEE Transactions on Image Processing.

[23]  Denis G. Pelli,et al.  ECVP '07 Abstracts , 2007, Perception.

[24]  Giuseppe Valenzise,et al.  Learning-based tone mapping operator for image matching , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[25]  Stefan Winkler,et al.  Analysis of Public Image and Video Databases for Quality Assessment , 2012, IEEE Journal of Selected Topics in Signal Processing.

[26]  Mark D. Fairchild,et al.  Color Appearance Models , 1997, Computer Vision, A Reference Guide.

[27]  Robert W. G. Hunt,et al.  The reproduction of colour , 1957 .

[28]  Rafal Mantiuk,et al.  Display adaptive tone mapping , 2008, SIGGRAPH 2008.

[29]  Michael S. Brown,et al.  Improving Color Reproduction Accuracy on Cameras , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Jing Li,et al.  Boosting paired comparison methodology in measuring visual discomfort of 3DTV: performances of three different designs , 2013, Electronic Imaging.

[31]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Erik Reinhard,et al.  Calibrated image appearance reproduction , 2012, ACM Trans. Graph..

[33]  Rafal Mantiuk,et al.  A practical guide and software for analysing pairwise comparison experiments , 2017, ArXiv.

[34]  Louis B. Rall,et al.  Automatic differentiation , 1981 .

[35]  Yi Li,et al.  Convolutional Neural Networks for No-Reference Image Quality Assessment , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Sabine Süsstrunk,et al.  Measuring colorfulness in natural images , 2003, IS&T/SPIE Electronic Imaging.

[37]  Tim Weyrich,et al.  A study of image colourfulness , 2014, CAe@Expressive.

[38]  François Pitié,et al.  Automated colour grading using colour distribution transfer , 2007, Comput. Vis. Image Underst..