Log file analysis for disengagement detection in e-Learning environments

Most e-Learning systems store data about the learner’s actions in log files, which give us detailed information about learner behaviour. Data mining and machine learning techniques can give meaning to these data and provide valuable information for learning improvement. One area that is of particular importance in the design of e-Learning systems is learner motivation as it is a key factor in the quality of learning and in the prevention of attrition. One aspect of motivation is engagement, a necessary condition for effective learning. Using data mining techniques for log file analysis, our research investigates the possibility of predicting users’ level of engagement, with a focus on disengaged learners. As demonstrated previously across two different e-Learning systems, HTML-Tutor and iHelp, disengagement can be predicted by monitoring the learners’ actions (e.g. reading pages and taking test/quizzes). In this paper we present the findings of three studies that refine this prediction approach. Results from the first study show that two additional reading speed attributes can increase the accuracy of prediction. The second study suggests that distinguishing between two different patterns of disengagement (spending a long time on a page/test and browsing quickly through pages/tests) may improve prediction in some cases. The third study demonstrates the influence of exploratory behaviour on prediction, as most users at the first login familiarize themselves with the system before starting to learn.

[1]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[2]  J. Keller Development and use of the ARCS model of instructional design , 1987 .

[3]  Neil T. Heffernan,et al.  Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.

[4]  Erik Duval,et al.  Creating New Learning Experiences on a Global Scale, Second European Conference on Technology Enhanced Learning, EC-TEL 2007, Crete, Greece, September 17-20, 2007, Proceedings , 2007, EC-TEL.

[5]  Gwo-Dong Chen,et al.  Promoting Motivation and Eliminating Disorientation for Web Based Courses by a Multi-User Game. , 1998 .

[6]  Mihaela Cocea,et al.  Can Log Files Analysis Estimate Learners' Level of Motivation? , 2006, LWA.

[7]  Ian Witten,et al.  Data Mining , 2000 .

[8]  A. Bandura Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[9]  Joseph S. Valacich,et al.  The Effects of Interruptions, Task Complexity, and Information Presentation on Computer-Supported Decision-Making Performance , 2003, Decis. Sci..

[10]  W. Lewis Johnson,et al.  Detecting the Learner's Motivational States in An Interactive Learning Environment , 2005, AIED.

[11]  Amy L. Baylor,et al.  Constructing Agents for Self-Learning: Animated Agents as Expressive Vehicles , 2003 .

[12]  Ryan Shaun Joazeiro de Baker,et al.  Detecting Student Misuse of Intelligent Tutoring Systems , 2004, Intelligent Tutoring Systems.

[13]  Helen Pain,et al.  Informing the Detection of the Students' Motivational State: An Empirical Study , 2002, Intelligent Tutoring Systems.

[14]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[15]  Lei Qu,et al.  Classifying Learner Engagement through Integration of Multiple Data Sources , 2006, AAAI.

[16]  Beverly Park Woolf,et al.  A Dynamic Mixture Model to Detect Student Motivation and Proficiency , 2006, AAAI.

[17]  Stephan Weibelzahl,et al.  Developing Adaptive Internet Based Courses with the Authoring System NetCoach , 2001, OHS-7/SC-3/AH-3.

[18]  Alexandre Martins,et al.  One for All and All in One , 1999 .

[19]  Mihaela Cocea,et al.  Assessment of Motivation in Online Learning Environments , 2006, AH.

[20]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[21]  P. Pintrich,et al.  Motivation in Education: Theory, Research, and Applications , 1995 .

[22]  Mihaela Cocea,et al.  Cross-System Validation of Engagement Prediction from Log Files , 2007, EC-TEL.

[23]  Margie Martyn,et al.  Clickers in the Classroom: An Active Learning Approach. , 2007 .

[24]  Beverly Park Woolf,et al.  Inferring learning and attitudes from a Bayesian Network of log file data , 2005, AIED.

[25]  Alexandre Martins,et al.  One for all and all in one: a learner modelling server in a multi-agent platform , 1999 .

[26]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[27]  Mihaela Cocea,et al.  Eliciting Motivation Knowledge from Log Files Towards Motivation Diagnosis for Adaptive Systems , 2007, User Modeling.

[28]  Peter Brusilovsky,et al.  Social Navigation Support in E-Learning : What are the Real Footprints ? , 2005 .

[29]  M. Csíkszentmihályi Finding Flow: The Psychology of Engagement with Everyday Life , 1997 .

[30]  Joseph E. Beck,et al.  Engagement tracing: using response times to model student disengagement , 2005, AIED.

[31]  Albert T. Corbett,et al.  Intelligent Tutoring Systems , 1985, Science.

[32]  Mark Stansfield,et al.  Journal of Information Technology Education Using Games-based Elearning Technologies in Overcoming Difficulties in Teaching Information Systems Game-based Technologies in Overcoming Difficulties in Teaching Information Systems Educational Issues Game-based Technologies in Overcoming Difficulties in , 2022 .

[33]  B. Shneiderman,et al.  Windows of opportunity in electronic classrooms , 1995, CACM.

[34]  Cheryl Campanella Bracken,et al.  Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects , 2005 .

[35]  Stephan Weibelzahl,et al.  Eliciting Adaptation Knowledge from On-Line Tutors to Increase Motivation , 2007, User Modeling.

[36]  Neil T. Heffernan,et al.  Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.