The Joint Multivariate Modeling of Multiple Mixed Response Sources: Relating Student Performances with Feedback Behavior

The present study concerns a Dutch computer-based assessment, which includes an assessment process about information literacy and a feedback process for students. The assessment is concerned with the measurement of skills in information literacy and the feedback process with item-based support to improve student learning. To analyze students’ feedback behavior (i.e. feedback use and attention time), test performance, and speed of working, a multivariate hierarchical latent variable model is proposed. The model can handle multivariate mixed responses from multiple sources related to different processes and comprehends multiple measurement components for responses and response times. A flexible within-subject latent variable structure is defined to explore multiple individual latent characteristics related to students’ test performance and feedback behavior. Main results of the computer-based assessment showed that feedback-information pages were less visited by well-performing students when they relate to easy items. Students’ attention paid to feedback was positively related to working speed but not to the propensity to use feedback.

[1]  J. Fox.  Bayesian Item Response Modeling: Theory and Applications , 2010 .

[2]  Theodorus Johannes Hendrikus Maria Eggen,et al.  The effectiveness of methods for providing written feedback through a computer-based assessment for learning: a systematic review , 2011 .

[3]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[4]  Wim J. van der Linden,et al.  Computerized Adaptive Testing With Item Cloning , 2003 .

[5]  Bernard P. Veldkamp,et al.  Attention paid to feedback provided by a computer-based assessment for learning on information literacy , 2011, Comput. Educ..

[6]  Robert J. Mislevy,et al.  Posterior Predictive Model Checking for Multidimensionality in Item Response Theory , 2006 .

[7]  Wim J. van der Linden,et al.  IRT Parameter Estimation With Response Times as Collateral Information , 2010 .

[8]  Bernard P. Veldkamp,et al.  Effects of feedback in a computer-based assessment for learning , 2012, Comput. Educ..

[9]  J. Albert.  Bayesian Estimation of Normal Ogive Item Response Curves Using Gibbs Sampling , 1992 .

[10]  Nicholas J. Belkin,et al.  Interactive information retrieval: history and background , 2011, Interactive Information Seeking, Behaviour and Retrieval.

[11]  Bradley A. Hanson,et al.  Development and Calibration of an Item Response Model That Incorporates Response Time , 2005 .

[12]  M. Knott,et al.  Generalized latent trait models , 2000 .

[13]  Martha C. Polson,et al.  Foundations of intelligent tutoring systems , 1988 .

[14]  A. Béguin,et al.  MCMC estimation and some model-fit analysis of multidimensional IRT models , 2001 .

[15]  W. D. Linden,et al.  Conceptual Issues in Response-Time Modeling. , 2009 .

[16]  V. Aleven,et al.  Help Seeking and Help Design in Interactive Learning Environments , 2003 .

[17]  Robert J. Mislevy,et al.  Putting ECD into Practice: The Interplay of Theory and Data in Evidence Models within a Digital Learning Environment , 2012, EDM 2012.

[18]  Willem J. van der Linden,et al.  A lognormal model for response times on test items , 2006 .

[19]  Cornelis A.W. Glas,et al.  Statistical Tests of Conditional Independence Between Responses and/or Response Times on Test Items , 2010 .

[20]  Brian W. Junker,et al.  The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data , 2002 .

[21]  Silvia Cagnone,et al.  A General Multivariate Latent Growth Model With Applications to Student Achievement , 2012 .

[22]  T. Ortner,et al.  A latent trait model for response times on tests employing the proportional hazards model. , 2012, The British journal of mathematical and statistical psychology.

[23]  Hal S. Stern,et al.  Posterior Predictive Assessment of Item Response Theory Models , 2006 .

[24]  Hua-Hua Chang,et al.  The linear transformation model with frailties for the analysis of item response times. , 2013, The British journal of mathematical and statistical psychology.

[25]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[26]  R. H. Klein Entink,et al.  A Multivariate Multilevel Approach to the Modeling of Accuracy and Speed of Test Takers , 2008, Psychometrika.

[27]  Assessing Fit of Unidimensional Graded Response Models Using Bayesian Methods , 2011 .

[28]  Wim J. van der Linden,et al.  Optimal Test Design With Rule-Based Item Generation , 2013 .

[29]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[30]  J. Templin,et al.  Measurement of psychological disorders using cognitive diagnosis models. , 2006, Psychological methods.

[31]  Cornelis A.W. Glas,et al.  Modeling Rule-Based Item Generation , 2011 .

[32]  Jean-Paul Fox,et al.  Bayesian Item Response Modeling , 2010 .

[33]  Saskia Brand-Gruwel,et al.  Information-problem solving: A review of problems students encounter and instructional solutions , 2008, Comput. Hum. Behav..

[34]  Daniel M. Bolt,et al.  A Comparison of Alternative Models for Testlets , 2006 .