Multimodal recognition of emotions in car environments

B Abstract — Within the last couple of years, automatic multimodal recognition of human emotions has gained a considerable interest from the research community. By taking into account more sources of information, the multimodal approaches allow for more reliable estimation of the human emotions. They increase the confidence of the results and decrease the level of ambiguity with respect to the emotions among the separate communication channels. This paper provides a thorough description of a bimodal emotion recognition system that uses face and speech analysis. Basically, we use hidden Markov models - HMMs to learn and to describe the temporal dynamics of the emotion clues in the visual and acoustic channels. We present the details of all steps involved in the analysis, from the preparation of the multimodal database and the feature extraction to the classification of six prototypic emotions. Apart from working with unimodal recognizers, we conduct experiments on both early fusion and decision level fusion of visual and audio features. The novelty of our approach consists of the dynamic modelling of emotions using hidden Markov models (HMMs) in combination with Local Binary Patterns (LBPs) (17) as visual features and mel-frequency cepstral coefficients (MFCCs) as audio features. In the same time, we propose a new method for visual feature selection based on the multi-class Adaboost.M2 classifier. A cross database method is employed to identify the set of most relevant features from a unimodal database and to proceed with applying it in the context of the multimodal setup. We report on the results we have achieved so far for the discussed models. The last part of the paper relates to conclusions and discussions on the possible ways to continue the research on the topic of multimodal emotion recognition.