An unobtrusive and multimodal approach for behavioral engagement detection of students

In this paper, we investigate detection of students’ behavioral engagement states (On-Task vs. Off-Task) in authentic classroom settings. We propose a multimodal detection approach, based on three unobtrusive modalities readily available in a 1:1 learning scenario where learning technologies are incorporated. These modalities are: (1)Appearance: upper-body video captured using a camera; (2) Context-Performance: students’ interaction and performance data related to learning content; and (3) Mouse: data related to mouse movements during learning process. For each modality, separate unimodal classifiers were trained, and decision-level fusion was applied to obtain final behavioral engagement states. We also analyzed each modality based on Instructional and Assessment sections separately (i.e., Instructional where a student is reading an article or watching an instructional video vs. Assessment where a student is solving exercises on the digital learning platform). We carried out various experiments on a dataset collected in an authentic classroom, where students used laptops equipped with cameras and they consumed learning content for Math on a digital learning platform. The dataset included multimodal data of 17 students who attended a Math course for 13 sessions (40 minutes each). The results indicate that it is beneficial to have separate classification pipelines for Instructional and Assessment sections: For Instructional, using only Appearance modality yields an F1-measure of 0.74, compared to fused performance of 0.70. For Assessment, fusing all three modalities (F1-measure of 0.89) provide a prominent improvement over the best performing unimodality (i.e., 0.81 for Appearance).

[1]  Kimberly A. Neuendorf,et al.  Reliability for Content Analysis , 2010 .

[2]  Ryan Shaun Joazeiro de Baker,et al.  Automatic Detection of Learning-Centered Affective States in the Wild , 2015, IUI.

[3]  Lisa Linnenbrink-Garcia,et al.  Academic Emotions and Student Engagement , 2012 .

[4]  Beverly Park Woolf,et al.  Affect-aware tutors: recognising and responding to student affect , 2009, Int. J. Learn. Technol..

[5]  Davis E. King,et al.  Dlib-ml: A Machine Learning Toolkit , 2009, J. Mach. Learn. Res..

[6]  Josephine Sullivan,et al.  One millisecond face alignment with an ensemble of regression trees , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  S. Christenson,et al.  Handbook of Research on Student Engagement , 2012 .

[8]  Eda Okur,et al.  Human Expert Labeling Process (HELP): Towards a Reliable Higher-order User State Labeling Process and Tool to Assess Student Engagement , 2017 .

[9]  Lisa M. Rossi,et al.  Student Off-Task Behavior in Computer-Based Learning in the Philippines: Comparison to Prior Research in the USA , 2013, Teachers College Record: The Voice of Scholarship in Education.

[10]  George D. Kuh,et al.  Student Engagement and Student Learning: Testing the Linkages* , 2006 .

[11]  Nese Alyüz,et al.  Behavioral Engagement Detection of Students in the Wild , 2017, AIED.

[12]  Berk Gökberk,et al.  Comparative Analysis of Decision-level Fusion Algorithms for 3D Face Recognition , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[13]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[14]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[15]  Chao Chen,et al.  Using Random Forest to Learn Imbalanced Data , 2004 .

[16]  Using Eye-Trackers to Study Student Attention in Physical Science Classes. , 2013 .

[17]  Ashish Kapoor,et al.  Multimodal affect recognition in learning environments , 2005, ACM Multimedia.

[18]  Andreas W. Kempa-Liehr,et al.  Distributed and parallel time series feature extraction for industrial big data applications , 2016, ArXiv.

[19]  Yueh-Min Huang,et al.  Developing a sensor-based learning concentration detection system , 2014 .

[20]  Nadia Mana,et al.  Towards the design of technology for measuring and capturing children's attention on e-learning tasks , 2013, IDC.

[21]  Kun Zhang,et al.  A hybrid intelligence-aided approach to affect-sensitive e-learning , 2014, Computing.

[22]  Supreeth M. Gowda,et al.  Affective States and State Tests: Investigating How Affect and Engagement during the School Year Predict End-of-Year Learning Outcomes , 2014, J. Learn. Anal..

[23]  Zehra Cataltepe,et al.  Learner Engagement Measurement and Classification in 1:1 Learning , 2014, 2014 13th International Conference on Machine Learning and Applications.

[24]  Bert Arnrich,et al.  Towards an Emotional Engagement Model: Can Affective States of a Learner be Automatically Detected in a 1: 1 Learning Scenario? , 2016, UMAP.

[25]  Zachary A. Pardos,et al.  Affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes , 2013, LAK '13.

[26]  Bert Arnrich,et al.  Semi-supervised model personalization for improved detection of learner's emotional engagement , 2016, ICMI.