Integration and Evaluation of a Matrix Factorization Sequencer in Large Commercial ITS

Correct evaluation of Machine Learning based sequencers require large data availability, large scale experiments and consideration of different evaluation measures. Such constraints make the construction of ad-hoc Intelligent Tutoring Systems (ITS) unfeasible and impose early integration in already existing ITS, which possesses a large amount of tasks to be sequenced. However, such systems were not designed to be combined with Machine Learning methods and require several adjustments. As a consequence more than a half of the components based on recommender technology are never evaluated with an online experiment. In this paper we show how we adapted a Matrix Factorization based performance predictor and a score based policy for task sequencing to be integrated in a commercial ITS with over 2000 tasks on 20 topics. We evaluated the experiment under different perspectives in comparison with the ITS sequencer designed by experts over the years. As a result we achieve same post-test results and outperform the current sequencer in the perceived experience questionnaire with almost no curriculum authoring effort. We also showed that the sequencer possess a better user modeling, better adapting to the knowledge acquisition rate of the students.

[1]  L. S. Vygotskiĭ,et al.  Mind in society : the development of higher psychological processes , 1978 .

[2]  Lars Schmidt-Thieme,et al.  Adaptive Content Sequencing without Domain Information , 2014, CSEDU.

[3]  Manolis Mavrikis,et al.  Matrix Factorization Feasibility for Sequencing and Adaptive Support in Intelligent Tutoring Systems , 2014, EDM.

[4]  Balaraman Ravindran,et al.  Personalized Intelligent Tutoring System Using Reinforcement Learning , 2011, FLAIRS.

[5]  Lars Schmidt-Thieme,et al.  Recommender system for predicting student performance , 2010, RecSysTEL@RecSys.

[6]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[7]  Katrien Verbert,et al.  Recommender Systems for Technology Enhanced Learning , 2014, Springer New York.

[8]  Kenneth R. Koedinger,et al.  Performance Factors Analysis - A New Alternative to Knowledge Tracing , 2009, AIED.

[9]  Lars Schmidt-Thieme,et al.  Using factorization machines for student modeling , 2012, UMAP Workshops.

[10]  Lars Schmidt-Thieme,et al.  Multimodal Affect Recognition for Adaptive Intelligent Tutoring Systems , 2014, EDM.

[11]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[12]  DIMITRIOS PIERRAKOS,et al.  User Modeling and User-Adapted Interaction , 1994, User Modeling and User-Adapted Interaction.

[13]  Kenneth R. Koedinger,et al.  Avoiding Problem Selection Thrashing with Conjunctive Knowledge Tracing , 2011, EDM.

[14]  Lars Schmidt-Thieme,et al.  Factorization Techniques for Predicting Student Performance , 2012 .

[15]  Peter Brusilovsky,et al.  Adaptive Hypermedia , 2001, User Modeling and User-Adapted Interaction.

[16]  Lars Schmidt-Thieme,et al.  Online-updating regularized kernel matrix factorization models for large-scale recommender systems , 2008, RecSys '08.

[17]  Zachary A. Pardos,et al.  KT-IDEM: introducing item difficulty to the knowledge tracing model , 2011, UMAP'11.

[18]  Zachary A. Pardos,et al.  Modeling Individualization in a Bayesian Networks Implementation of Knowledge Tracing , 2010, UMAP.

[19]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[20]  Neil T. Heffernan,et al.  The Student Skill Model , 2012, ITS.

[21]  Lars Schmidt-Thieme,et al.  Feature Analysis for Affect Recognition Supporting Task Sequencing in Adaptive Intelligent Tutoring Systems , 2014, EC-TEL.

[22]  Kurt VanLehn,et al.  Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies , 2011, User Modeling and User-Adapted Interaction.

[23]  Vincent Aleven,et al.  More Accurate Student Modeling through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing , 2008, Intelligent Tutoring Systems.

[24]  Joseph E. Beck,et al.  ADVISOR: A Machine Learning Architecture for Intelligent Tutor Construction , 2000, AAAI/IAAI.

[25]  Alfred Kobsa.  User Modeling and User-Adapted Interaction , 2005, User Modeling and User-Adapted Interaction.