Video Emotion Recognition with Concept Selection

Understanding video content is a challenging problem in many applications, especially for emotion analysis. Diverse and complicated video contents are the major obstacles in video emotion understanding. In this paper, we propose a modality fusion framework to combine concept and content features from action, scene and object models. We conduct concept selection to investigate the relations between high-level concept features and emotions. The discriminative concepts play important roles in emotion recognition. Fusing different modality of content features further improve the performance. The extensive experiments show the state-of-the-art results on two challenging video emotion benchmarks.

[1]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Luc Van Gool,et al.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition , 2016, ECCV.

[3]  Tao Chen,et al.  DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks , 2014, ArXiv.

[4]  Boyang Li,et al.  Video Emotion Recognition with Transferred Deep Feature Encodings , 2016, ICMR.

[5]  Luc Van Gool,et al.  Transferring Deep Object and Scene Representations for Event Recognition in Still Images , 2017, International Journal of Computer Vision.

[6]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[7]  Allan Hanbury,et al.  Affective image classification using features inspired by psychology and art theory , 2010, ACM Multimedia.

[8]  Bolei Zhou,et al.  Object Detectors Emerge in Deep Scene CNNs , 2014, ICLR.

[9]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[10]  Andrew Zisserman,et al.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Nicu Sebe,et al.  Emotional valence categorization using holistic image features , 2008, 2008 15th IEEE International Conference on Image Processing.

[12]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Boyang Li,et al.  Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization , 2015, IEEE Transactions on Affective Computing.

[14]  James Ze Wang,et al.  On shape and the computability of emotions , 2012, ACM Multimedia.

[15]  Bolei Zhou,et al.  Places: A 10 Million Image Database for Scene Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Xiangyang Xue,et al.  Predicting Emotions in User-Generated Videos , 2014, AAAI.

[17]  Haimin Zhang,et al.  Recognition of Emotions in User-Generated Videos With Kernelized Features , 2018, IEEE Transactions on Multimedia.

[18]  P. Ekman.  An argument for basic emotions , 1992 .

[19]  Xiangjian He,et al.  Hierarchical affective content analysis in arousal and valence dimensions , 2013, Signal Process..

[20]  Heng Wang,et al.  Dense Dilated Network for Few Shot Action Recognition , 2018, ICMR.

[21]  Wei Liu,et al.  DeepProduct: Mobile Product Search With Portable Deep Features , 2018, ACM Trans. Multim. Comput. Commun. Appl..

[22]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.

[23]  Shih-Fu Chang,et al.  Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification , 2017, IEEE Transactions on Multimedia.

[24]  Jiebo Luo,et al.  Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks , 2015, AAAI.

[25]  Li-Jia Li,et al.  Visual Sentiment Prediction with Deep Convolutional Neural Networks , 2014, ArXiv.

[26]  Hang-Bong Kang,et al.  Affective content detection using HMMs , 2003, ACM Multimedia.

[27]  Chen Chen,et al.  Emotion in Context: Deep Semantic Feature Fusion for Video Emotion Recognition , 2016, ACM Multimedia.

[28]  Robert Plutchik,et al.  EMOTIONS AND PSYCHOTHERAPY: A PSYCHOEVOLUTIONARY PERSPECTIVE , 1990 .