dAFM: Fusing Psychometric and Connectionist Modeling for Q-Matrix Refinement.

We introduce a model which combines principles from psychometric and connectionist paradigms to allow direct Q-matrix refinement via backpropagation. We call this model dAFM, based on augmentation of the original Additive Factors Model (AFM), whose calculations and constraints we show can be exactly replicated within the framework of neural networks. In order to parameterize the Q-matrix definition in the model, the associations between questions and knowledge components (KC) need to be represented by adjustable weights. Furthermore, student KC opportunity counts, instead of serving as fixed inputs, need to be calculated dynamically as the Q-matrix changes during training. We describe our solutions to these two modeling challenges and evaluate several variants of our fully realized model on datasets from the Cognitive Tutor and ASSISTments. We compare learning the Q-matrix from scratch vs. refining an expert specified KC model and evaluate various procedures for refinement. In our quantitative predictive analysis, we find that dAFM learns a better generalizing Q-matrix than the original expert model in all our primary datasets. Using a development set, we also find that the dAFM Q-matrix is superior to KC representations extracted from trained Deep Knowledge Tracing and skip-gram models. Examples are shown of questions whose fit was improved by dAFM with depictions of their original and refined KC associations. We consistently find in our experiments that our dAFM variant which attempted to learn the Q-matrix from scratch underperformed models which started with an expert defined Q-matrix that was then refined. This observation continues a theme in EDM of utility found in the enduring value of expert domain knowledge enhanced through data-driven refinement.

[1]  K. Koedinger,et al.  Automating Cognitive Model Improvement by A * Search and Logistic Regression , 2005 .

[2]  Michel C. Desmarais.  Conditions for Effectively Deriving a Q-Matrix from Data with Non-negative Matrix Factorization. Best Paper Award , 2011, EDM.

[3]  Kenneth R. Koedinger,et al.  Performance Factors Analysis - A New Alternative to Knowledge Tracing , 2009, AIED.

[4]  Albert T. Corbett,et al.  Cognitive Computer Tutors: Solving the Two-Sigma Problem , 2001, User Modeling.

[5]  Zachary A. Pardos,et al.  The 2010 KDD Cup Competition Dataset: Engaging the machine learning community in predictive learning analytics , 2016, J. Learn. Anal..

[6]  Peter Brusilovsky,et al.  General Features in Knowledge Tracing to Model Multiple Subskills, Temporal Item Response Theory, and Expert Knowledge , 2014, EDM.

[7]  Michel C. Desmarais,et al.  A Matrix Factorization Method for Mapping Items to Skills and for Enhancing Expert-Based Q-Matrices , 2013, AIED.

[8]  Neil T. Heffernan,et al.  The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learning and Teaching , 2014, International Journal of Artificial Intelligence in Education.

[9]  Mitchell J. Nathan,et al.  Expert Blind Spot : When Content Knowledge Eclipses Pedagogical Content Knowledge , 2001 .

[10]  Antonija Mitrovic,et al.  Evaluating and improving adaptive educational systems with learning curves , 2011, User Modeling and User-Adapted Interaction.

[11]  Jack Mostow,et al.  Dynamic Cognitive Tracing: Towards Unified Discovery of Student and Cognitive Models , 2012, EDM.

[12]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[13]  Chia-Yi Chiu.  Statistical Refinement of the Q-Matrix in Cognitive Diagnosis , 2013 .

[14]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[15]  Zachary A Pardos,et al.  Big data in education and the models that love them , 2017, Current Opinion in Behavioral Sciences.

[16]  Kelly G. Shaver,et al.  The attribution of blame : causality, responsibility, and blameworthiness , 1985 .

[17]  Kenneth R. Koedinger,et al.  Is Over Practice Necessary? - Improving Learning Efficiency with the Cognitive Tutor through Educational Data Mining , 2007, AIED.

[18]  Kenneth R. Koedinger,et al.  Individualized Bayesian Knowledge Tracing Models , 2013, AIED.

[19]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[20]  Ran Liu,et al.  Modeling common misconceptions in learning process data , 2016, LAK.

[21]  Leonidas J. Guibas,et al.  Deep Knowledge Tracing , 2015, NIPS.

[22]  Richard Scheines,et al.  Discovering Prerequisite Relationships Among Knowledge Components , 2014, EDM.

[23]  Kenneth R. Koedinger,et al.  Automated Student Model Improvement , 2012, EDM.

[24]  Sara Fripp.  A learning curve. , 2014, Midwives.

[25]  Neil T. Heffernan,et al.  Automatic and Semi-Automatic Skill Coding With a View Towards Supporting On-Line Assessment , 2005, AIED.

[26]  Jenna Burrell,et al.  How the machine ‘thinks’: Understanding opacity in machine learning algorithms , 2016 .

[27]  David F. Feldon,et al.  Cognitive task analysis , 2009 .

[28]  Kenneth R. Koedinger,et al.  Learning Factors Transfer Analysis: Using Learning Curve Analysis to Automatically Generate Domain Models , 2009, EDM.

[29]  J. Fodor,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[30]  Zachary A. Pardos,et al.  Imputing KCs with Representations of Problem Content and Context , 2017, UMAP.

[31]  Albert T. Corbett,et al.  Cognitive Tutor: Applied research in mathematics education , 2007, Psychonomic bulletin & review.

[32]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[33]  G. Rasch.  On General Laws and the Meaning of Measurement in Psychology , 1961 .

[34]  Yi Sun,et al.  Alternating Recursive Method for Q-matrix Learning , 2014, EDM.

[35]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[36]  Kenneth R. Koedinger,et al.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement , 2006, Intelligent Tutoring Systems.

[37]  Jingchen Liu,et al.  Data-Driven Learning of Q-Matrix , 2012, Applied psychological measurement.

[38]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[39]  Sidney D'Mello,et al.  Data mining and education. , 2015, Wiley interdisciplinary reviews. Cognitive science.

[40]  Omer Levy,et al.  Linguistic Regularities in Sparse and Explicit Word Representations , 2014, CoNLL.

[41]  Nils J. Nilsson,et al.  A Formal Basis for the Heuristic Determination of Minimum Cost Paths , 1968, IEEE Trans. Syst. Sci. Cybern..

[42]  Yann LeCun,et al.  The Loss Surfaces of Multilayer Networks , 2014, AISTATS.

[43]  Radek Pelánek,et al.  Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques , 2017, User Modeling and User-Adapted Interaction.

[44]  Shaghayegh Sahebi,et al.  Tensor Factorization for Student Modeling and Performance Prediction in Unstructured Domain , 2016, EDM.

[45]  Jin Tian,et al.  Joint Discovery of Skill Prerequisite Graphs and Student Models , 2016, EDM.

[46]  Radek Pelánek,et al.  Metrics for Evaluation of Student Models , 2015, EDM.

[47]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[48]  Robert V. Lindsey,et al.  Incorporating Latent Factors Into Knowledge Tracing To Predict Individual Differences In Learning , 2013 .