Automatically detected acoustic landmarks for assessing natural emotion from speech

This Master thesis focuses on the field of emotion recognition from speech, for which the interest has grown considerably during the past ten years with more than 100 papers per year since 2004. Today, sophisticated and relevant methods to extract features from speech permitting to analyse emotions are available. Although these methods give convincing results on acted emotions, such emotions are not really natural and used techniques are thus not applicable in the real world. Current techniques mainly focus on acoustic speech features, e.g., pitch, energy, voice quality, and lead to poor performances when they are applied on spontaneous and natural recording of speech. Thus, the present challenge for this field resides in the search for relevant techniques permitting to a robust emotion recognition from spontaneous speech recordings. A possible solution would consist in exploiting linguistic features that are being used for the emotion recognition task. However, to enabling future non-human interaction partners, the use of linguistic features has to be done in an automatic way, as it is the case for the analysis of the acoustic features. Until now, only few studies focused on emotion recognition through linguistic features. Some used automatic speech recognition (ASR) systems output to extract features rather than manual annotation of the data. Results show that experiments based on ASR output lead to lower performances than experiments based on manual annotation. This comes from the fact that ASR is likely to introduce inaccuracies in the transcriptions, because the emotion introduces significant variabilities. This Master thesis investigates new techniques for linguistic features extraction from speech with the aim to achieve emotion recognition without using ASR systems. Thus, instead of using word transcription from ASR systems, our technique exploits acoustic landmarks, i.e., events correlated with changes in speech production and perception, which are automatically detectable in the speech signal. We compared emotion recognition performance of manually transcribed words and automatically detected acoustic landmarks on the same corpus (SEMAINE), using state-of-the-art methods for features extraction, e.g., Bag of words and n-grams. Results show that considering simple landmarks as voiced/unvoiced segments of speech already leads to better performance than using manually transcribed words. When considering vowels/consonant landmarks or the p-center landmark alone, i.e., rhythmic events, no improvement can be observed. Whereas best scores are obtained on the fusion of voiced/unvoiced landmarks with p-centers. Although the difference between scores obtained on words and those from the fusion of voiced/unvoiced landmarks and p-centers is not significant, this work shows the interest of working with acoustic landmarks. Indeed, by automatically extracting linguistic features through the acoustic signal, we are able to overcome the need of working with ASR systems, while improving scores.