An Upgrading Procedure for Adaptive Assessment of Knowledge

In knowledge space theory, existing adaptive assessment procedures can only be applied when suitable estimates of their parameters are available. In this paper, an iterative procedure is proposed, which upgrades its parameters with the increasing number of assessments. The first assessments are run using parameter values that favor accuracy over efficiency. Subsequent assessments are run using new parameter values estimated on the incomplete response patterns from previous assessments. Parameter estimation is carried out through a new probabilistic model for missing-at-random data. Two simulation studies show that, with the increasing number of assessments, the performance of the proposed procedure approaches that of gold standards.

[1]  Luca Stefanutti,et al.  Modeling missing data in knowledge space theory. , 2015, Psychological methods.

[2]  Jean-Claude Falmagne,et al.  The Assessment of Knowledge, in Theory and in Practice , 2006, ICFCA.

[3]  Klaus Korossy,et al.  Modeling Knowledge as Competence and Performance , 1999 .

[4]  Jean-Paul Doignon.  Knowledge Spaces and Skill Assignments , 1994 .

[5]  Jean-Claude Falmagne,et al.  Learning Spaces: Interdisciplinary Applied Mathematics , 2010 .

[6]  Cosyn,et al.  A Practical Procedure to Build a Knowledge Structure. , 2000, Journal of mathematical psychology.

[7]  Mathieu Koppen,et al.  Introduction to knowledge spaces: How to build, test, and search them , 1990 .

[8]  Mathieu Koppen,et al.  How to build a knowledge space by querying an expert , 1990 .

[9]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[10]  Ying Cheng.  When Cognitive Diagnosis Meets Computerized Adaptive Testing: CD-CAT , 2009 .

[11]  Jimmy de la Torre,et al.  Model Evaluation and Multiple Strategies in Cognitive Diagnosis: An Analysis of Fraction Subtraction Data , 2008 .

[12]  Luca Stefanutti,et al.  Uncovering the Best Skill Multimap by Constraining the Error Probabilities of the Gain-Loss Model , 2012 .

[13]  Eric Cosyn,et al.  Note on two necessary and sufficient axioms for a well-graded knowledge space , 2009 .

[14]  Luca Stefanutti,et al.  Assessing learning processes with the gain-loss model , 2011, Behavior research methods.

[15]  Marc Harper,et al.  ALEKS-based Placement at the University of Illinois , 2013, Knowledge Spaces, Applications in Education.

[16]  Jean-Paul Doignon,et al.  A class of stochastic procedures for the assessment of knowledge. , 1988 .

[17]  Cord Hockemeyer.  A Comparison of Non-Deterministic Procedures for the Adaptive Assessment of Knowledge , 2002 .

[18]  Cord Hockemeyer,et al.  Automata for the Assessment of Knowledge , 2001, IEEE Trans. Knowl. Data Eng..

[19]  J. Pannekoek,et al.  Bootstrapping Goodness-of-Fit Measures in Categorical Data Analysis , 1996 .

[20]  Jürgen Heller,et al.  On the Link between Cognitive Diagnostic Models and Knowledge Space Theory , 2015, Psychometrika.

[21]  Hua-Hua Chang,et al.  Combining computer adaptive testing technology with cognitively diagnostic assessment , 2008, Behavior research methods.

[22]  J. Templin,et al.  The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model , 2008 .

[23]  Ivo Düntsch,et al.  Skills and knowledge structures , 1995 .

[24]  Luca Stefanutti,et al.  The Gain-Loss Model: A Probabilistic Skill Multimap Model for Assessing Learning Processes , 2010 .

[25]  Pao-Kuei Wu,et al.  MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING , 1996 .

[26]  Jean-Claude Falmagne,et al.  Spaces for the Assessment of Knowledge , 1985, Int. J. Man Mach. Stud..

[27]  Luca Stefanutti,et al.  A Procedure for Identifying the Best Skill Multimap in the Gain-Loss Model , 2013, Electron. Notes Discret. Math..

[28]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[29]  D. Rubin.  INFERENCE AND MISSING DATA , 1975 .

[30]  Luca Stefanutti,et al.  Recovering a Probabilistic Knowledge Structure by Constraining its Parameter Space , 2009 .

[31]  Jürgen Heller,et al.  Exploiting Prior Information in Stochastic Knowledge Assessment , 2012 .

[32]  Rebecca Holman,et al.  Modelling non-ignorable missing-data mechanisms with item response theory models. , 2005, The British journal of mathematical and statistical psychology.

[33]  Jean-Paul Doignon,et al.  A Markovian procedure for assessing the state of a system , 1988 .

[34]  Jean-Claude Falmagne,et al.  Languages for the assessment of knowledge , 1986 .

[35]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[36]  Jean-Claude Falmagne,et al.  Knowledge spaces , 1998 .

[37]  Alan Huebner,et al.  An Overview of Recent Developments in Cognitive Diagnostic Computer Adaptive Assessments. , 2010 .

[38]  Dimitra Dodou,et al.  Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon , 2010 .

[39]  Cornelia E. Dowling.  Applying the basis of a knowledge space for controlling the questioning of an expert , 1993 .

[40]  Jürgen Heller,et al.  Erratum to: On the Link between Cognitive Diagnostic Models and Knowledge Space Theory , 2016, Psychometrika.

[41]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[42]  Dietrich Albert,et al.  Skills, Competencies and Knowledge Structures , 2013, Knowledge Spaces, Applications in Education.

[43]  Mathieu Koppen,et al.  Extracting human expertise for constructing knowledge spaces: an algorithm , 1993 .

[44]  Ivo Düntsch,et al.  Skill set analysis in knowledge structures. , 2002, The British journal of mathematical and statistical psychology.

[45]  Curtis Tatsuoka,et al.  Data analytic methods for latent partially ordered classification models , 2002 .

[46]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .