Estimation of ability from homework items when there are missing and/or multiple attempts

Scoring of student item response data from online courses and especially massively open online courses (MOOCs) is complicated by two challenges, potentially large amounts of missing data and allowances for multiple attempts to answer. Approaches to ability estimation with respect to both of these issues are considered using data from a large-enrollment electrical engineering MOOC. The allowance of unlimited multiple attempts sets up a range of observed score and latent-variable approaches to scoring the constructed response homework. With respect to missing data, two classical approaches are discussed, treating omitted items as incorrect or missing at random (MAR). These treatments turn out to have slightly different interpretations depending on the scoring model. In all, twelve different homework scores are proposed based on combinations of scoring model and missing data handling. The scores are computed and correlations between each score and the final exam score are compared, with attention to different populations of course participants.

[1]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[2]  Michael W. Browne,et al.  Psychometrics , 2000, The SAGE Encyclopedia of Research Design.

[3]  Frederic M. Lord,et al.  Estimation of latent ability and item parameters when there are omitted responses , 1974 .

[4]  Dimitrios Rizopoulos.  ltm: An R Package for Latent Variable Modeling and Item Response Theory Analyses , 2006 .

[5]  Yoav Bergner,et al.  Who does what in a massive open online course? , 2014, Commun. ACM.

[6]  L. Andries van der Ark,et al.  Relationships and Properties of Polytomous Item Response Theory Models , 2001 .

[7]  Dean Stamoulis,et al.  What to Assess , 2009 .

[8]  Russell G. Almond,et al.  You Can't Fatten A Hog by Weighing It - Or Can You? Evaluating an Assessment for Learning System Called ACED , 2008, Int. J. Artif. Intell. Educ..

[9]  D. Bates,et al.  Linear Mixed-Effects Models using 'Eigen' and S4 , 2015 .

[10]  Frank B. Baker,et al.  Item Response Theory : Parameter Estimation Techniques, Second Edition , 2004 .

[11]  Dimitris Rizopoulos,et al.  ltm: An R Package for Latent Variable Modeling and Item Response Analysis , 2006 .

[12]  V. Shute.  Focus on Formative Feedback , 2007 .

[13]  M. Davier,et al.  Modeling Nonignorable Missing Data with Item Response Theory (IRT). Research Report. ETS RR-10-11. , 2010 .

[14]  Andrew D. Ho,et al.  Changing “Course”: Reconceptualizing Educational Variables for Massive Open Online Courses , 2014 .

[15]  R. J. Mokken,et al.  Handbook of modern item response theory , 1997 .

[16]  D. Rubin.  INFERENCE AND MISSING DATA , 1975 .

[17]  Steffi Pohl,et al.  Dealing With Omitted and Not-Reached Items in Competence Tests , 2014 .

[18]  G. Brosvic,et al.  Immediate Feedback during Academic Testing , 2001, Psychological reports.

[19]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[20]  G. Tutz.  Sequential item response models with an ordered response , 1990 .

[21]  Yigal Attali,et al.  Immediate Feedback and Opportunity to Revise Answers , 2011 .

[22]  Paul De Boeck,et al.  IRTrees: Tree-Based Item Response Models of the GLMM Family , 2012 .

[23]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1968 .

[24]  Donald E. Powers,et al.  Immediate Feedback and Opportunity to Revise Answers to Open-Ended Questions , 2010 .

[25]  V. Shute.  Focus on Formative Feedback. Research Report. ETS RR-07-11. , 2007 .

[26]  Pao-Kuei Wu,et al.  MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING , 1996 .

[27]  Mark D. Reckase,et al.  Item Response Theory: Parameter Estimation Techniques , 1998 .