Performance comparison of item-to-item skills models with the IRT single latent trait model

Assessing a learner's mastery of a set of skills is a fundamental issue in intelligent learning environments. We compare the predictive performance of two approaches for training a learner model with domain data. One is based on the principle of building the model solely from observable data items, such as exercises or test items. Skills modelling is not part of the training phase, but instead dealt with at later stage. The other approach incorporates a single latent skill in the model. We compare the capacity of both approaches to accurately predict item outcome (binary success or failure) from a subset of item outcomes. Three types of item-to-item models based on standard Bayesian modeling algorithms are tested: (1) Naive Bayes, (2) Tree-Augmented Naive Bayes (TAN), and (3) a K2 Bayesian Classifier. Their performance is compared to the widely used IRT-2PL approach which incorporates a single latent skill. The results show that the item-to-item approaches perform as well, or better than the IRT-2PL approach over 4 widely different data sets, but the differences vary considerably among the data sets. We discuss the implications of these results and the issues relating to the practical use of item-to-item models.

[1]  Jean-Claude Falmagne,et al.  Knowledge spaces , 1998 .

[2]  R. J. Mokken,et al.  Handbook of modern item response theory , 1997 .

[3]  Ricardo Conejo,et al.  Introducing Prerequisite Relations in a Multi-layered Bayesian Student Model , 2005, User Modeling.

[4]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[5]  Bernhard Ganter,et al.  Formal Concept Analysis , 2013 .

[6]  Dimitrios Rizopoulos.  ltm: An R Package for Latent Variable Modeling and Item Response Theory Analyses , 2006 .

[7]  Rebecca Nugent,et al.  A Comparison of Student Skill Knowledge Estimates , 2009, EDM.

[8]  Kenneth R. Koedinger,et al.  Learning Factors Transfer Analysis: Using Learning Curve Analysis to Automatically Generate Domain Models , 2009, EDM.

[9]  Karl Rihaczek,et al.  1. WHAT IS DATA MINING? , 2019, Data Mining for the Social Sciences.

[10]  Chao-Lin Liu,et al.  A Simulation-Based Experience in Learning Structures of Bayesian Networks to Represent How Students Learn Composite Concepts , 2008, Int. J. Artif. Intell. Educ..

[11]  Michel C. Desmarais,et al.  Learned student models with item to item knowledge structures , 2006, User Modeling and User-Adapted Interaction.

[12]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[13]  Dietrich Albert,et al.  RATH - A Relational Adaptive Tutoring Hypertext WWW-Environment Based on Knowledge Space Theory , 1997 .

[14]  Paul Brna,et al.  User Modeling 2005, 10th International Conference, UM 2005, Edinburgh, Scotland, UK, July 24-29, 2005, Proceedings , 2005, User Modeling.

[15]  Kurt VanLehn,et al.  Student Modeling from Conversational Test Data: A Bayesian Approach Without Priors , 1998, Intelligent Tutoring Systems.

[16]  F. Baker,et al.  Item response theory : parameter estimation techniques , 1993 .

[17]  Thomas Lengauer,et al.  ROCR: visualizing classifier performance in R , 2005, Bioinform..

[18]  Dietrich Albert,et al.  Competence-based Knowledge Structures for Personalised Learning , 2005, ELeGI Conference.

[19]  Tiffany Barnes,et al.  Extracting Student Models for Intelligent Tutoring Systems , 2007, AAAI.

[20]  Michel C. Desmarais,et al.  A Bayesian Student Model without Hidden Nodes and its Comparison with Item Response Theory , 2005, Int. J. Artif. Intell. Educ..

[21]  Cristina Conati,et al.  Unsupervised and supervised machine learning in user modeling for intelligent learning environments , 2007, IUI '07.

[22]  Jirí Vomlel,et al.  Bayesian Networks In Educational Testing , 2002, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[23]  Dimitris Rizopoulos,et al.  ltm: An R Package for Latent Variable Modeling and Item Response Analysis , 2006 .

[24]  Jean-Paul Doignon,et al.  The assessment of knowledge, in theory and in practice , 2003, IEMC '03 Proceedings. Managing Technologically Driven Organizations: The Human Side of Innovation and Change (IEEE Cat. No.03CH37502).

[25]  Dietrich Albert,et al.  Applying competence structures for peer tutor recommendations in CSCL environments , 2004, IEEE International Conference on Advanced Learning Technologies, 2004. Proceedings..

[26]  Cristina Conati,et al.  Using Bayesian Networks to Manage Uncertainty in Student Modeling , 2002, User Modeling and User-Adapted Interaction.

[27]  Thomas Lengauer,et al.  Data and text mining ROCR : visualizing classifier performance in R , 2005 .