Affective image classification using features inspired by psychology and art theory

Images can affect people on an emotional level. Since the emotions that arise in the viewer of an image are highly subjective, they are rarely indexed. However there are situations when it would be helpful if images could be retrieved based on their emotional content. We investigate and develop methods to extract and combine low-level features that represent the emotional content of an image, and use these for image emotion classification. Specifically, we exploit theoretical and empirical concepts from psychology and art theory to extract image features that are specific to the domain of artworks with emotional expression. For testing and training, we use three data sets: the International Affective Picture System (IAPS); a set of artistic photography from a photo sharing site (to investigate whether the conscious use of colors and textures displayed by the artists improves the classification); and a set of peer rated abstract paintings to investigate the influence of the features and ratings on pictures without contextual content. Improved classification results are obtained on the International Affective Picture System (IAPS), compared to state of the art work.

[1]  J. M. Kittross.  The measurement of meaning , 1959 .

[2]  R. Arnheim.  Art and Visual Perception, a Psychology of the Creative Eye , 1967 .

[3]  J. Itten.  The art of color : the subjective experience and objective rationale of color , 1973 .

[4]  Hideyuki Tamura,et al.  Textural Features Corresponding to Visual Perception , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[5]  P. Ekman,et al.  DIFFERENCES Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion , 2004 .

[6]  Linda G. Shapiro,et al.  Computer and Robot Vision , 1991 .

[7]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[8]  P. Valdez,et al.  Effects of color on emotions. , 1994, Journal of experimental psychology. General.

[9]  Emanuele Trucco,et al.  Computer and Robot Vision , 1995 .

[10]  Masafumi Hagiwara,et al.  Image query by impression words-the IQI system , 1998 .

[11]  Alberto Del Bimbo,et al.  Semantics in Visual Information Retrieval , 1999, IEEE Multim..

[12]  Alberto Del Bimbo,et al.  Image retrieval by color semantics , 1999, Multimedia Systems.

[13]  Nadia Bianchi-Berthouze,et al.  K-DIME: An Affective Image Filtering System , 2003, IEEE Multim..

[14]  Sung-Bae Cho,et al.  Emotional image and musical information retrieval with interactive genetic algorithm , 2004, Proc. IEEE.

[15]  L. Ou,et al.  A study of colour emotion and colour preference. Part III: Colour preference modeling , 2004 .

[16]  L. Ou,et al.  A study of colour emotion and colour preference. Part I: Colour emotions for single colours , 2004 .

[17]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[18]  Sam J. Maglio,et al.  Emotional category data on images from the international affective picture system , 2005, Behavior research methods.

[19]  Beatriz Marcotegui,et al.  Fast Implementation of Waterfall Based on Graphs , 2005, ISMM.

[20]  Alan Hanjalic,et al.  Affective video content representation and modeling , 2005, IEEE Transactions on Multimedia.

[21]  P. Lang.  International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[22]  Changle Zhou,et al.  Content-Based Affective Image Classification and Retrieval Using Support Vector Machines , 2005, ACII.

[23]  Wei-Ning Wang,et al.  Image emotional semantic query based on color semantic description , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[24]  Loong Fah Cheong,et al.  Affective understanding in film , 2006, IEEE Trans. Circuits Syst. Video Technol..

[25]  A. Hanjalic,et al.  Extracting moods from pictures and sounds: towards truly personalized TV , 2006, IEEE Signal Processing Magazine.

[26]  Shengming Jiang,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, SMC.

[27]  James Ze Wang,et al.  Studying Aesthetics in Photographic Images Using a Computational Approach , 2006, ECCV.

[28]  Nicu Sebe,et al.  Content-based multimedia information retrieval: State of the art and challenges , 2006, TOMCCAP.

[29]  Cordelia Schmid,et al.  Learning Color Names from Real-World Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Qianhua He,et al.  A survey on emotional semantic image retrieval , 2008, 2008 15th IEEE International Conference on Image Processing.

[31]  Allan Hanbury,et al.  Constructing cylindrical coordinate colour spaces , 2008, Pattern Recognit. Lett..

[32]  Nicu Sebe,et al.  Emotional valence categorization using holistic image features , 2008, 2008 15th IEEE International Conference on Image Processing.

[33]  Julian Stöttinger,et al.  Color-based and context-aware skin detection for online video annotation , 2009, 2009 IEEE International Workshop on Multimedia Signal Processing.

[34]  Nicu Sebe,et al.  Translating Journalists' Requirements into Features for Image Search , 2009, 2009 15th International Conference on Virtual Systems and Multimedia.

[35]  S. R. Jammalamadaka,et al.  Directional Statistics, I , 2011 .