Emotion in Context: Deep Semantic Feature Fusion for Video Emotion Recognition

Human emotions demonstrate high correlations with certain events that consist of the interaction of objects, and are usually constrained by particular scenes. In this paper, we exploit these abundant context clues for video emotion recognition. We first compute event, object and scene scores with state-of-the-art detectors based on deep neural networks. The extracted high-level features serve as effective contextual information demonstrating what is occurring in the video, and are further integrated in a context fusion network to generate a unified representation aiming to bridge the affective gap. The contribution of this paper is three-fold: (a) we are the first to incorporate event as context for emotion recognition, and we demonstrate its superiority for emotion understanding, (b) we utilize a context fusion network to exploit a comprehensive set of high-level semantics features as contextual clues and (c) the proposed framework can realize recognition in real-time and achieves state-of-the-art performance on two challenging emotion recognition benchmarks, 50.6 on VideoEmotion and 51.8 on Ekman.