Introducing the Geneva Multimodal Emotion Portrayal (GEMEP) corpus

In this chapter we outline the requirements for a systematic corpus of actor portrayals and describe the development, recording, editing, and validating of a major new corpus, the Geneva Multimodal Emotion Portrayal (GEMEP). This corpus consists of more than 7,000 audio-video emotion portrayals, representing 18 emotions (including rarely studied subtle emotions), portrayed by 10 professional actors who were coached by a professional director. The portrayals are recorded with optimal digital quality in multiple modalities, using both pseudo linguistic utterances and affect bursts. In addition, the corpus includes stimuli with systematically varied intensity levels, as well as instances of masked expressions. From the total corpus, 1,260 portrayals were selected and submitted to a first rating procedure in different modalities to establish validity in terms of inter-judge reliability and recognition accuracy. The results show that the portrayed expressions are recognized by lay judges with an accuracy level that, in the case of all emotions, largely exceeded chance and that compares very favorably with published tests of emotion recognition that use highly selected stimulus sets. The portrayals also reach very satisfactory levels of inter-rater reliability for category judgments and ratings of believability and intensity of the portrayals. The validity of the corpus is further confirmed by replicating results in earlier work on the role of expression modality and the corresponding communication channel for cue utilization in emotion recognition. We show that, as expected, the highest accuracy results if both auditory and visual information (voice, face, and gestures) is available, but that sizeable accuracy is achieved even when only one modality is available. The video modality is slightly superior to the audio modality, probably reflecting the fact that facial and gestural cues are more discrete and iconic than vocal cues. However, there are important interactions between emotion and modality, as particular emotions seem to be preferentially communicated by visual or audio cues. The results also raise important issues concerning the relationships