Exploring Principles-of-Art Features For Image Emotion Recognition

Emotions can be evoked in humans by images. Most previous works on image emotion analysis mainly used the elements-of-art-based low-level visual features. However, these features are vulnerable and not invariant to the different arrangements of elements. In this paper, we investigate the concept of principles-of-art and its influence on image emotions. Principles-of-art-based emotion features (PAEF) are extracted to classify and score image emotions for understanding the relationship between artistic principles and emotions. PAEF are the unified combination of representation features derived from different principles, including balance, emphasis, harmony, variety, gradation, and movement. Experiments on the International Affective Picture System (IAPS), a set of artistic photography and a set of peer rated abstract paintings, demonstrate the superiority of PAEF for affective image classification and regression (with about 5% improvement on classification accuracy and 0.2 decrease in mean squared error), as compared to the state-of-the-art approaches. We then utilize PAEF to analyze the emotions of master paintings, with promising results.

[1]  J. Itten.  The art of color : the subjective experience and objective rationale of color , 1973 .

[2]  P. Lang.  International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[3]  Ling-Yu Duan,et al.  Hierarchical movie affective content analysis based on arousal and valence features , 2008, ACM Multimedia.

[4]  Rongrong Ji,et al.  Photo assessment based on computational visual attention model , 2009, ACM Multimedia.

[5]  Vicente Ordonez,et al.  High level describable attributes for predicting aesthetics and interestingness , 2011, CVPR 2011.

[6]  Xuelong Li,et al.  Visual-Textual Joint Relevance Learning for Tag-Based Social Image Search , 2013, IEEE Transactions on Image Processing.

[7]  James Ze Wang,et al.  On shape and the computability of emotions , 2012, ACM Multimedia.

[8]  P. Lang,et al.  International Affective Picture System (IAPS): Instruction Manual and Affective Ratings (Tech. Rep. No. A-4) , 1999 .

[9]  Hongxun Yao,et al.  Affective Image Retrieval via Multi-Graph Learning , 2014, ACM Multimedia.

[10]  Daniel Cohen-Or,et al.  Optimizing Photo Composition , 2010, Comput. Graph. Forum.

[11]  James Ze Wang,et al.  Algorithmic inferencing of aesthetics and emotion in natural images: An exposition , 2008, 2008 15th IEEE International Conference on Image Processing.

[12]  Claus Bahlmann,et al.  Fast radial symmetry detection under affine transformations , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Jiebo Luo,et al.  Aesthetics and Emotions in Images , 2011, IEEE Signal Processing Magazine.

[14]  Wei Zhang,et al.  Emotion based image musicalization , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[15]  Yu Ying-lin,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[16]  Dana H. Ballard,et al.  Generalizing the Hough transform to detect arbitrary shapes , 1981, Pattern Recognit..

[17]  Riccardo Leonardi,et al.  A Connotative Space for Supporting Movie Affective Recommendation , 2011, IEEE Transactions on Multimedia.

[18]  Allan Hanbury,et al.  Affective image classification using features inspired by psychology and art theory , 2010, ACM Multimedia.

[19]  M. Kass,et al.  Smoothed local histogram filters , 2010, ACM Trans. Graph..

[20]  Bing Li,et al.  Context-aware affective images classification based on bilayer sparse representation , 2012, ACM Multimedia.

[21]  Sam J. Maglio,et al.  Emotional category data on images from the international affective picture system , 2005, Behavior research methods.

[22]  Bates Lowry,et al.  The visual experience , 1961 .

[23]  Rongrong Ji,et al.  What are we looking for: Towards statistical modeling of saccadic eye movements and visual saliency , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Hatice Gunes,et al.  A multi-layer hybrid framework for dimensional emotion classification , 2011, ACM Multimedia.

[25]  Alexander Zelinsky,et al.  Fast Radial Symmetry for Detecting Points of Interest , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Rongrong Ji,et al.  Video indexing and recommendation based on affective analysis of viewers , 2011, MM '11.

[27]  Mohan S. Kankanhalli,et al.  Affect-based adaptive presentation of home videos , 2011, ACM Multimedia.

[28]  Jan-Olof Eklundh,et al.  Detecting Symmetry and Symmetric Constellations of Features , 2006, ECCV.

[29]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[30]  Cordelia Schmid,et al.  Learning Color Names from Real-World Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Xiaoou Tang,et al.  Photo and Video Quality Evaluation: Focusing on the Subject , 2008, ECCV.

[32]  Nicu Sebe,et al.  Emotional valence categorization using holistic image features , 2008, 2008 15th IEEE International Conference on Image Processing.

[33]  A. Hanjalic,et al.  Extracting moods from pictures and sounds: towards truly personalized TV , 2006, IEEE Signal Processing Magazine.

[34]  H. Schlosberg.  Three dimensions of emotion. , 1954, Psychological review.

[35]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.

[36]  Hang-Bong Kang,et al.  Affective content detection using HMMs , 2003, ACM Multimedia.

[37]  Kiyoharu Aizawa,et al.  Affective Audio-Visual Words and Latent Topic Driving Model for Realizing Movie Affective Scene Classification , 2010, IEEE Transactions on Multimedia.

[38]  Reiner Lenz,et al.  Color Based Bags-of-Emotions , 2009, CAIP.

[39]  Jie Tang,et al.  Can we understand van gogh's mood?: learning to infer affects from images in social networks , 2012, ACM Multimedia.

[40]  Shengming Jiang,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, SMC.

[41]  R. Arnheim.  Art and visual perception: A psychology of the creative eye, New version , 1955 .

[42]  Li Xu,et al.  Structure extraction from texture via relative total variation , 2012, ACM Trans. Graph..

[43]  Bernard Mérialdo,et al.  Enhancing Semantic Features with Compositional Analysis for Scene Recognition , 2012, ECCV Workshops.

[44]  Jan C. van Gemert,et al.  Exploiting photographic style for category-level image classification by generalizing the spatial pyramid , 2011, ICMR.

[45]  A. Philip McMahon,et al.  The Principles of Art , 1939 .

[46]  R. Arnheim.  Art and Visual Perception, a Psychology of the Creative Eye , 1967 .