Specifying and Refining a Measurement Model for a Computer-Based Interactive Assessment

The challenges of modeling students' performance in computer-based interactive assessments include accounting for multiple aspects of knowledge and skill that arise in different situations and the conditional dependencies among multiple aspects of performance. This article describes a Bayesian approach to modeling and estimating cognitive models in such situations, both in terms of statistical machinery and actual instrument development. The method taps the knowledge of experts to provide initial estimates for the probabilistic relations among the variables in a multivariate latent variable model and refines these estimates using Markov chain Monte Carlo procedures. This process is illustrated in the context of Networking Performance Skill System (NetPASS), a computer-based interactive assessment in the domain of computer networking. We describe a parameterization of the relations in NetPASS via an ordered polytomous item-response model and detail the updating of the model with observed data via Bayesian statistical procedures ultimately being provided by Markov chain Monte Carlo estimation.

[1]  W Edwards.  Hailfinder. Tools for and experiences with Bayesian normative modeling. , 1998, The American psychologist.

[2]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[3]  Andrew Gelman,et al.  General methods for monitoring convergence of iterative simulations , 1998 .

[4]  Sylvia Richardson,et al.  Inference and monitoring convergence , 1995 .

[5]  Mark D. Reckase,et al.  The Difficulty of Test Items That Measure More Than One Ability , 1985 .

[6]  ANTON K. FORMANN,et al.  Structural Latent Class Models , 1998 .

[7]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[8]  David J. Spiegelhalter,et al.  Bayesian analysis in expert systems , 1993 .

[9]  R. Hambleton,et al.  Item Response Theory , 1984, The History of Educational Measurement.

[10]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1968 .

[11]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[12]  David J. Spiegelhalter,et al.  Introducing Markov chain Monte Carlo , 1995 .

[13]  Sarah F. Demark,et al.  Using Statistical Natural Language Processing for Understanding Complex Responses to Free-Response Tasks , 2004 .

[14]  R. Mislevy.  Evidence and inference in educational assessment , 1994 .

[15]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[16]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[17]  David A. Schum,et al.  Evidence and inference for the intelligence analyst , 1987 .

[18]  Anton K. Formann,et al.  Constrained latent class models: Theory and applications , 1985 .

[19]  Eric T. Bradlow,et al.  A Bayesian random effects model for testlets , 1999 .

[20]  Russell G. Almond,et al.  Modeling Conditional Probabilities in Complex Educational Assessments. CSE Technical Report. , 2002 .

[21]  Peter Green,et al.  Markov chain Monte Carlo in Practice , 1996 .

[22]  Russell G. Almond,et al.  On the Structure of Educational Assessments, CSE Technical Report. , 2003 .

[23]  D. Lindley,et al.  Bayes Estimates for the Linear Model , 1972 .

[24]  R. C. Durfee,et al.  MULTIPLE FACTOR ANALYSIS. , 1967 .

[25]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[26]  David Andrich,et al.  An extension of the rasch model for ratings providing both location and dispersion parameters , 1982 .

[27]  Stephen P. Brooks,et al.  Markov chain Monte Carlo method and its application , 1998 .

[28]  Russell G. Almond,et al.  Graphical Models and Computerized Adaptive Testing , 1998 .