SPRITE : A Data-Driven Response Model For Multiple Choice Questions

Item response theory (IRT) models, in their most basic, dichotomous form, model a set of binary-valued (correct/incorrect) responses from individuals to items/questions. These models are ubiquitous in computer-based learning analytics and assessment applications, because they enable the inference of latent student abilities/respondent traits. Since the option a student selects on a multiple-choice question (either the correct response or one of the incorrect, distractor responses) contains more information regarding the student’s ability than a simple binary-valued grade, polytomous IRT models have been developed to cover the cases of unordered (i.e., categorical) options and strictly ordered (i.e., ordinal) options. However, in many real-world educational scenarios, the various distractor options in a multiple-choice question are neither categorical, since they are incorrect to varying degrees, nor ordinal, since they are not strictly ordered. Moreover, this (partial) ordering information might not be known a priori, inhibiting the application of existing polytomous IRT models to practical scenarios. In this work, we propose the SPRITE (short for stochastic polytomous response item) model, a novel IRT extension for multiple-choice questions with unknown, partially ordered options. SPRITE improves substantially over existing IRT models in that it (i) learns the (partial) ordering of the options directly from student response data, (ii) produces interpretable model parameters, and (iii) outperforms existing approaches on predicting unobserved student responses on multiple real-world educational datasets.