Accurate Estimation of Time-on-Task While Programming

In a recent study, students were periodically prompted to self-report engagement while working on computer programming assignments in a CS1 course. A regression model predicting time-on-task was proposed. While it was a significant improvement over ad-hoc estimation techniques, the study nevertheless suffered from lack of error analysis, lack of comparison with existing methods, subtle complications in prompting students, and small sample size. In this paper we report results from a study with an increased number of student participants and modified prompting scheme intended to better capture natural student behavior. Furthermore, we perform a cross-validation analysis on our refined regression model and present the resulting error bounds. We compare with threshold approaches and find that, in at least one context, a simple 5-minute threshold of inactivity is a reasonable estimate for whether a student is on-task or not. We show that our approach to modeling student engagement while programming is robust and suitable for identification of students in need of intervention, understanding engagement behavior, and estimating time taken on programming assignments.