PARAMETER RECOVERY STUDIES WITH A DIAGNOSTIC BAYESIAN NETWORK MODEL

This paper describes a Bayesian network model for a candidate assessment design that had four proficiency variables and 48 tasks with 3–12 observable outcome variables per task and scale anchors to identify the location of the subscales. The domain experts’ view of the relationship among proficiencies and tasks established a complex prior distribution over 585 parameters. Markov Chain Monte Carlo (MCMC) estimation recovered the parameters of data simulated from the expert model. The sample size and the strength of the prior had only a modest effect on parameter recovery, but did affect the standard error of estimated parameters. Finally, an identifiability issue involving relabeling of proficiency states and permutations of the matrixes is addressed in the context of this study.

[1]  Eric T. Bradlow,et al.  Testlet Response Theory and Its Applications , 2007 .

[2]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[3]  Russell G. Almond,et al.  Bayes Nets in Educational Assessment: Where the Numbers Come From , 1999, UAI.

[4]  Russell G. Almond,et al.  Modeling Diagnostic Assessments with Bayesian Networks , 2007 .

[5]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[6]  Gregory M. Provan,et al.  Why is diagnosis using belief networks insensitive to imprecision in probabilities? , 1996, UAI.

[7]  Russell G. Almond,et al.  Models for Conditional Probability Tables in Educational Assessment , 2001, AISTATS.

[8]  S. Frühwirth-Schnatter.  Markov chain Monte Carlo Estimation of Classical and Dynamic Switching and Mixture Models , 2001 .

[9]  David J. Spiegelhalter,et al.  Sequential updating of conditional probabilities on directed graphical structures , 1990, Networks.

[10]  Russell G. Almond,et al.  Graphical Models and Computerized Adaptive Testing , 1999 .

[11]  Irvin R. Katz.  Testing Information Literacy in Digital Environments: ETS's iSkills Assessment , 2007 .

[12]  Russell G. Almond,et al.  Bayesian Network Models for Local Dependence Among Observable Outcome Variables , 2006 .

[13]  Russell G. Almond,et al.  "I Can Name that Bayesian Network in Two Matrixes!" , 2007, BMA.

[14]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[15]  Andrew Gelman,et al.  General methods for monitoring convergence of iterative simulations , 1998 .

[16]  Ron Miller,et al.  Enterprise digital asset management system pilot: Lessons learned , 2007 .