Fully Automatic Analysis of Engagement and Its Relationship to Personality in Human-Robot Interactions

Engagement is crucial to designing intelligent systems that can adapt to the characteristics of their users. This paper focuses on the automatic analysis and classification of engagement based on humans’ and robot’s personality profiles in a triadic human–human–robot interaction setting. More explicitly, we present a study that involves two participants interacting with a humanoid robot, and investigate how participants’ personalities can be used together with the robot’s personality to predict the engagement state of each participant. The fully automatic system is first trained to predict the Big Five personality traits of each participant by extracting individual and interpersonal features from their nonverbal behavioural cues. Second, the output of the personality prediction system is used as an input to the engagement classification system. Third, we focus on the concept of “group engagement”, which we define as the collective engagement of the participants with the robot, and analyze the impact of similar and dissimilar personalities on the engagement classification. Our experimental results show that: 1) using the automatically predicted personality labels for engagement classification yields an F-measure on par with using the manually annotated personality labels, demonstrating the effectiveness of the automatic personality prediction module proposed; 2) using the individual and interpersonal features without utilizing personality information is not sufficient for engagement classification, instead incorporating the participants and robots personalities with individual/interpersonal features increases engagement classification performance; and 3) the best classification performance is achieved when the participants and the robot are extroverted, while the worst results are obtained when all are introverted.