Reading Students' Multiple Mental States in Conversation from Facial and Heart Rate Cues

Students’ mental states have been widely acknowledged as crucial components for inferring their learning processes and are closely linked with learning outcomes. Understanding students’ complex mental states including concentration, confusion, frustration, and boredom in teacher-student conversation could benefit a human teacher’s perceptual and real-time decision-making capability in providing personalized and adaptive support in coaching activities. Many lines of research have explored the automatic measurement of students’ mental states in pre-designed human-computer tasks. It still remains a challenge to detect the complex mental states of students in real teacher-student conversation. In this study, we made such an attempt by describing a system for predicting the complex mental states of students from multiple perspectives: facial and physiological (heart rate) cues in real student-teacher conversation scenarios. We developed an advanced multi-sensor-based system and applied it in small-scale meetings to collect students’ multimodal conversation data. We demonstrate a multimodal analysis framework. Machine learning models were built by using extracted interpretable proxy features at a fine-grained level to validate their predictive ability regarding students’ multiple mental states. Our results provide evidence of the potential value of fusing multimodal data to understand students’ multiple mental states in real-world student-teacher conversation.