Multimodal Database of Emotional Speech, Video and Gestures

People express emotions through different modalities. Integration of verbal and non-verbal communication channels creates a system in which the message is easier to understand. Expanding the focus to several expression forms can facilitate research on emotion recognition as well as human-machine interaction. In this article, the authors present a Polish emotional database composed of three modalities: facial expressions, body movement and gestures, and speech. The corpora contains recordings registered in studio conditions, acted out by 16 professional actors (8 male and 8 female). The data is labeled with six basic emotions categories, according to Ekman’s emotion categories. To check the quality of performance, all recordings are evaluated by experts and volunteers. The database is available to academic community and might be useful in the study on audio-visual emotion recognition.

[1]  R. Cowie,et al.  A new emotion database: considerations, sources and scope , 2000 .

[2]  P. Wilson,et al.  The Nature of Emotions , 2012 .

[3]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[4]  Dirk Kraft,et al.  Emotion Recognition through Body Language using RGB-D Sensor , 2016, VISIGRAPP.

[5]  A. Pease,et al.  The Definitive Book of Body Language , 2004 .

[6]  Richa Singh,et al.  RGB-D Face Recognition With Texture and Attribute Features , 2014, IEEE Transactions on Information Forensics and Security.

[7]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[8]  Sergio Escalera,et al.  Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[9]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[10]  B. Gelder,et al.  Why bodies? Twelve reasons for including bodily expressions in affective neuroscience , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[11]  Petros Daras,et al.  Multimodal affective state recognition in serious games applications , 2016, 2016 IEEE International Conference on Imaging Systems and Techniques (IST).

[12]  R I Hg,et al.  An RGB-D Database Using Microsoft's Kinect for Windows for Face Detection , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[13]  E. Scilingo,et al.  Arousal and Valence Recognition of Affective Sounds Based on Electrodermal Activity , 2017, IEEE Sensors Journal.

[14]  Gholamreza Anbarjafari,et al.  Efficiency of chosen speech descriptors in relation to emotion recognition , 2017, EURASIP Journal on Audio, Speech, and Music Processing.

[15]  Michal Niedzwiecki,et al.  Hand Body Language Gesture Recognition Based on Signals From Specialized Glove and Machine Learning Algorithms , 2016, IEEE Transactions on Industrial Informatics.

[16]  Sergio Escalera,et al.  Deep Multimodal Pain Recognition: A Database and Comparison of Spatio-Temporal Visual Modalities , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[17]  Shaun J. Canavan,et al.  BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database , 2014, Image Vis. Comput..

[18]  Jean-Luc Dugelay,et al.  KinectFaceDB: A Kinect Database for Face Recognition , 2014, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[19]  Sergio Escalera,et al.  Survey on Emotional Body Gesture Recognition , 2018, IEEE Transactions on Affective Computing.

[20]  R. Plutchik.  Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice , 2016 .

[21]  Gholamreza Anbarjafari,et al.  Vocal-based emotion recognition using random forests and decision tree , 2017, Int. J. Speech Technol..

[22]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[23]  Yong Du,et al.  Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks , 2017, IEEE Transactions on Image Processing.

[24]  Adam Kendon,et al.  THE STUDY OF GESTURE: SOME REMARKS ON ITS HISTORY , 1983 .

[25]  Sethuraman Panchanathan,et al.  Multimodal emotion recognition using deep learning architectures , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[26]  Martin Buss,et al.  Feature Extraction and Selection for Emotion Recognition from EEG , 2014, IEEE Transactions on Affective Computing.

[27]  Astrid Paeschke,et al.  A database of German emotional speech , 2005, INTERSPEECH.

[28]  Gholamreza Anbarjafari,et al.  3D Scanning: A Comprehensive Survey , 2018, ArXiv.

[29]  Nicu Sebe,et al.  A Quality Adaptive Multimodal Affect Recognition System for User-Centric Multimedia Indexing , 2016, ICMR.

[30]  P. Ekman.  Universals and cultural differences in facial expressions of emotion. , 1972 .

[31]  Sergio Escalera,et al.  Results and Analysis of ChaLearn LAP Multi-modal Isolated and Continuous Gesture Recognition, and Real Versus Fake Expressed Emotions Challenges , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[32]  J. P. Foley,et al.  Gesture and Environment , 1942 .

[33]  Mihai Gavrilescu,et al.  Recognizing emotions from videos by studying facial expressions, body postures and hand gestures , 2015, 2015 23rd Telecommunications Forum Telfor (TELFOR).

[34]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[35]  Daniel McDuff,et al.  Real-time inference of mental states from facial expressions and upper body gestures , 2011, Face and Gesture 2011.

[36]  Sazali Yaacob,et al.  Emotion recognition from facial EMG signals using higher order statistics and principal component analysis , 2014 .

[37]  Gholamreza Anbarjafari,et al.  SASE: RGB-Depth Database for Human Head Pose Estimation , 2016, ECCV Workshops.

[38]  Harriet Oster,et al.  Japanese and American infants' responses to arm restraint. , 1992 .