A Multi-component CNN-RNN Approach for Dimensional Emotion Recognition in-the-wild

This paper presents our approach to the One-Minute Gradual-Emotion Recognition (OMG-Emotion) Challenge, focusing on dimensional emotion recognition through visual analysis of the provided emotion videos. The approach is based on a Convolutional and Recurrent (CNN-RNN) deep neural architecture we have developed for the relevant large AffWild Emotion Database. We extended and adapted this architecture, by letting a combination of multiple features generated in the CNN component be explored by RNN subnets. Our target has been to obtain best performance on the OMG-Emotion visual validation data set, while learning the respective visual training data set. Extended experimentation has led to best architectures for the estimation of the values of the valence and arousal emotion dimensions over these data sets.

[1]  Dimitrios Kollias,et al.  Face Behavior à la carte: Expressions, Affect and Action Units in a Single Network , 2019, ArXiv.

[2]  Jesse Hoey,et al.  From individual to group-level emotion recognition: EmotiW 5.0 , 2017, ICMI.

[3]  Stefanos Zafeiriou,et al.  Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition , 2018, ArXiv.

[4]  Andreas Stafylopatis,et al.  Interweaving deep learning and semantic techniques for emotion analysis in human-machine interaction , 2015, 2015 10th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP).

[5]  Stefanos Zafeiriou,et al.  Training Deep Neural Networks with Different Datasets In-the-wild: The Emotion Recognition Paradigm , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[6]  Stefanos Zafeiriou,et al.  Photorealistic Facial Synthesis in the Dimensional Affect Space , 2018, ECCV Workshops.

[7]  Stefanos Zafeiriou,et al.  A Multi-Task Learning & Generation Framework: Valence-Arousal, Action Units & Primary Expressions , 2018, ArXiv.

[8]  Andreas Stafylopatis,et al.  On line emotion detection using retrainable deep neural networks , 2016, 2016 IEEE Symposium Series on Computational Intelligence (SSCI).

[9]  Dimitrios Kollias,et al.  Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace , 2019, BMVC.

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Stefan Wermter,et al.  The OMG-Emotion Behavior Dataset , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[12]  Luc Van Gool,et al.  Face Detection without Bells and Whistles , 2014, ECCV.

[13]  Guoying Zhao,et al.  Recognition of Affect in the Wild Using Deep Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[14]  Andreas Stafylopatis,et al.  Adaptation and contextualization of deep neural network models , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[15]  Yannis Avrithis,et al.  Broadcast news parsing using visual cues: a robust face detection approach , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[16]  Guoying Zhao,et al.  Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Guoying Zhao,et al.  Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond , 2018, International Journal of Computer Vision.

[18]  Andreas Stafylopatis,et al.  Deep neural architectures for prediction in healthcare , 2017, Complex & Intelligent Systems.

[19]  Fabien Ringeval,et al.  AVEC 2017: Real-life Depression, and Affect Recognition Workshop and Challenge , 2017, AVEC@ACM Multimedia.

[20]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Stefanos Zafeiriou,et al.  Generating faces for affect analysis , 2018, ArXiv.

[22]  Dimitrios Kollias,et al.  Exploiting multi-CNN features in CNN-RNN based Dimensional Emotion Recognition on the OMG in-the-wild Dataset , 2019, ArXiv.