A Multi-Stream Recurrent Neural Network for Social Role Detection in Multiparty Interactions

Understanding multiparty human interaction dynamics is a challenging problem involving multiple data modalities and complex ordered interactions between multiple people. We propose a unified framework that integrates synchronized video, audio, and text streams from four people to capture the interaction dynamics in natural group meetings. We focus on estimating the dynamic social role of the meeting participants, i.e., Protagonist, Neutral, Supporter, or Gatekeeper. Our key innovation is to incorporate both co-occurrence features and successive occurrence features in thin time windows to better describe the behavior of a target participant and his/her responses from others, using a multi-stream recurrent neural network. We evaluate our algorithm on the widely-used AMI corpus and achieve state-of-the-art accuracy of 78% for automatic dynamic social role detection. We further investigate the importance of different video and audio features for estimating social roles.