Affective Image Content Analysis: A Comprehensive Survey

Images can convey rich semantics and induce strong emotions in viewers. Recently, with the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this paper, we review the stateof-the-art methods comprehensively with respect to two main challenges – affective gap and perception subjectivity. We begin with an introduction to the key emotion representation models that have been widely employed in AICA. Available existing datasets for performing evaluation are briefly described. We then summarize and compare the representative approaches on emotion feature extraction, personalized emotion prediction, and emotion distribution learning. Finally, we discuss some future research directions.

[1]  Joonwhoan Lee,et al.  Fuzzy Similarity-Based Emotional Classification of Color Images , 2011, IEEE Transactions on Multimedia.

[2]  Min Xu,et al.  Learning Multi-level Deep Representations for Image Emotion Classification , 2016, Neural Processing Letters.

[3]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.

[4]  No Value,et al.  IEEE International Conference on Image Processing , 2003 .

[5]  Junqing Yu,et al.  An improved valence-arousal emotion space for video affective content representation and recognition , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[6]  Yue Gao,et al.  Learning Visual Emotion Distributions via Multi-Modal Features Fusion , 2017, ACM Multimedia.

[7]  James Ze Wang,et al.  On shape and the computability of emotions , 2012, ACM Multimedia.

[8]  Marco Aiello,et al.  AAAI Conference on Artificial Intelligence , 2011, AAAI Conference on Artificial Intelligence.

[9]  H. Damasio,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence: Special Issue on Perceptual Organization in Computer Vision , 1998 .

[10]  Ming-Hsuan Yang,et al.  Retrieving and Classifying Affective Images via Deep Metric Learning , 2018, AAAI.

[11]  Tsuhan Chen,et al.  A mixed bag of emotions: Model, predict, and transfer emotion distributions , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Lexing Xie,et al.  SentiCap: Generating Image Descriptions with Sentiments , 2015, AAAI.

[13]  Juan-Zi Li,et al.  How Do Your Friends on Social Media Disclose Your Emotions? , 2014, AAAI.

[14]  Jiebo Luo,et al.  Robust Visual-Textual Sentiment Analysis: When Attention meets Tree-structured Recursive Neural Networks , 2016, ACM Multimedia.

[15]  Yue Gao,et al.  Personality-Aware Personalized Emotion Recognition from Physiological Signals , 2018, IJCAI.

[16]  Jufeng Yang,et al.  Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network , 2017, IJCAI.

[17]  Youbao Tang,et al.  Discrete Probability Distribution Prediction of Image Emotions with Shared Sparse Learning , 2020, IEEE Transactions on Affective Computing.

[18]  Nicu Sebe,et al.  Who's Afraid of Itten: Using the Art Theory of Color Combination to Analyze Emotions in Abstract Paintings , 2015, ACM Multimedia.

[19]  Jiebo Luo,et al.  Sentribute: image sentiment analysis from a mid-level perspective , 2013, WISDOM '13.

[20]  P. Ekman.  An argument for basic emotions , 1992 .

[21]  Jiebo Luo,et al.  Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark , 2016, AAAI.

[22]  Sam J. Maglio,et al.  Emotional category data on images from the international affective picture system , 2005, Behavior research methods.

[23]  Bing Li,et al.  Context-aware affective images classification based on bilayer sparse representation , 2012, ACM Multimedia.

[24]  Yue Gao,et al.  Predicting Personalized Image Emotion Perceptions in Social Networks , 2018, IEEE Transactions on Affective Computing.

[25]  K. Scherer,et al.  The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance , 2011, Behavior research methods.

[26]  Lianhong Cai,et al.  Interpretable aesthetic features for affective image classification , 2013, 2013 IEEE International Conference on Image Processing.

[27]  Yue Gao,et al.  Exploring Principles-of-Art Features For Image Emotion Recognition , 2014, ACM Multimedia.

[28]  A. Hanjalic,et al.  Extracting moods from pictures and sounds: towards truly personalized TV , 2006, IEEE Signal Processing Magazine.

[29]  IEEE conference on computer vision and pattern recognition , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[30]  R. Plutchik.  Emotion, a psychoevolutionary synthesis , 1980 .

[31]  Yue Gao,et al.  Real-Time Multimedia Social Event Detection in Microblog , 2018, IEEE Transactions on Cybernetics.

[32]  Yue Gao,et al.  Predicting Microblog Sentiments via Weakly Supervised Multimodal Deep Learning , 2018, IEEE Transactions on Multimedia.

[33]  Zhi-Hua Zhou,et al.  Facial Age Estimation by Learning from Label Distributions , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  S. Grossberg,et al.  Psychological Review , 2003 .

[35]  Qingming Huang,et al.  Dependency Exploitation: A Unified CNN-RNN Approach for Visual Emotion Recognition , 2017, IJCAI.

[36]  P. Lang.  International Affective Picture System (IAPS) : Technical Manual and Affective Ratings , 1995 .

[37]  Rama Chellappa,et al.  Visual Domain Adaptation: A survey of recent advances , 2015, IEEE Signal Processing Magazine.