Emotion recognition by assisted learning with convolutional neural networks

Abstract Image emotion is the emotion hidden in or passed by a particular image. In this paper, a novel convolutional neural network is proposed to predict the emotion from an image. The proposed model consists of two parts: a binary positive-or-negative emotion classification network and a deep network for specific emotion recognition. During the network training, an assisted learning strategy is introduced to boost the recognition performance. Experimental results demonstrate that the proposed network is capable of extracting active level features and achieves significant gains in emotion recognition accuracy.

[1]  Y-Lan Boureau,et al.  Learning Convolutional Feature Hierarchies for Visual Recognition , 2010, NIPS.

[2]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[3]  Gareth J. F. Jones,et al.  Affect-based indexing and retrieval of films , 2005, MULTIMEDIA '05.

[4]  Deng Cai,et al.  Tracking people in RGBD videos using deep learning and motion clues , 2016, Neurocomputing.

[5]  Xiaogang Wang,et al.  Visual Tracking with Fully Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  James Ze Wang,et al.  On shape and the computability of emotions , 2012, ACM Multimedia.

[7]  Yu Ying-lin,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[8]  P. Valdez,et al.  Effects of color on emotions. , 1994, Journal of experimental psychology. General.

[9]  Jiebo Luo,et al.  Aesthetics and Emotions in Images , 2011, IEEE Signal Processing Magazine.

[10]  Yann LeCun,et al.  Convolutional networks and applications in vision , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[11]  Hatice Gunes,et al.  A multi-layer hybrid framework for dimensional emotion classification , 2011, ACM Multimedia.

[12]  Emmanuel Dellandréa,et al.  Classification of affective semantics in images based on discrete and dimensional models of emotions , 2010, 2010 International Workshop on Content Based Multimedia Indexing (CBMI).

[13]  Xiaolong Wang,et al.  Active deep learning method for semi-supervised sentiment classification , 2013, Neurocomputing.

[14]  Jiebo Luo,et al.  Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks , 2015, AAAI.

[15]  Yue Gao,et al.  Exploring Principles-of-Art Features For Image Emotion Recognition , 2014, ACM Multimedia.

[16]  Nicu Sebe,et al.  Emotional valence categorization using holistic image features , 2008, 2008 15th IEEE International Conference on Image Processing.

[17]  Allan Hanbury,et al.  Affective image classification using features inspired by psychology and art theory , 2010, ACM Multimedia.

[18]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[19]  Sam J. Maglio,et al.  Emotional category data on images from the international affective picture system , 2005, Behavior research methods.

[20]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[21]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[22]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Dumitru Erhan,et al.  Deep Neural Networks for Object Detection , 2013, NIPS.

[24]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.

[25]  M. Bradley,et al.  Emotion, Motivation, and Anxiety: Brain Mechanisms and Psychophysiology the Motivational Organization of Emotion Patterns of Human Emotion Emotion and Perception the Psychophysiology of Picture Processing Neural Imaging: Motivation in the Visual Cortex Motivational Circuits in the Brain , 2022 .

[26]  Alberto Del Bimbo,et al.  Semantics in Visual Information Retrieval , 1999, IEEE Multim..

[27]  Luca Maria Gambardella,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Flexible, High Performance Convolutional Neural Networks for Image Classification , 2022 .

[28]  Jiebo Luo,et al.  Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark , 2016, AAAI.

[29]  Xavier Giró-i-Nieto,et al.  From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction , 2016, Image Vis. Comput..