Towards an Emotional Engagement Model: Can Affective States of a Learner be Automatically Detected in a 1: 1 Learning Scenario?

Existing Intelligent Tutoring Systems (ITSs) are unable to track affective states of learners. In this paper, we focus on the problem of emotional engagement, and propose to detect important affective states (i.e., ‘Satisfied’, ‘Bored’, and ‘Confused’) of a learner in real time. We collected 210 hours of data from 20 students through authentic classroom pilots. The data included information from two modalities: (1) appearance which is collected from the camera, and (2) context-performance that is derived from the content platform. In this paper, data from nine students who attended the learning sessions twice a week are analyzed. We trained separate classifiers for different modalities (appearance and context-performance), and for different types of learning sections (instructional and assessment). The results show that different sources of information are generically better representatives of engagement at different sections: For instructional sections, generic appearance classifier yields higher accuracy (55.79%); whereas context-performance classifier is more accurate for assessment sections (63.41%). Moreover, the results indicate that expression of engagement is person-specific through both of these sources, and personalized engagement models perform more accurately: When personspecific data are added to the training set, on instructional sections, 85.44% and 96.13% accuracies are achieved for appearance and context-performance, respectively. For assessment sections, the accuracies are 75.25% (appearance) and 90.24% (contextperformance). When only person-specific data are employed during training, similar accuracies are achieved even with very limited data. CCS Concepts • Human-centered computing➝Human-computer interaction • Human-centered computing➝Personal computing.