Computational Modeling of Human Multimodal Language : The MOSEI Dataset and Interpretable Dynamic Fusion

Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities. Comprehending multimodal language requires not only the modeling of interactions within each modality (intramodal interactions), but more importantly the interactions between modalities (crossmodal interactions). Modeling these interactions lie at the core of multimodal language analysis. From a resource perspective, there is a genuine need for large scale datasets that allow for in-depth studies of human multimodal language. In this paper we introduce Multimodal Opinion Sentiment and Emotion Intensity (MOSEI), the largest dataset for multimodal sentiment analysis and emotion recognition. In addition, we propose a novel multimodal fusion technique called the Graph Memory Fusion Network (GMFN) that dynamically fuses modalities in a hierarchical manner. Using data from MOSEI and GMFN, we conduct experiments to investigate the hierarchical interactions between modalities in human multimodal language. Unlike previously proposed fusion techniques, GMFN is highly interpretable and achieves superior performance when compared to the previous state of the art, demonstrating that GMFN is highly suitable for multimodal language analysis.