A Signal Detection Model for Multiple-Choice Exams

A model for multiple-choice exams is developed from a signal-detection perspective. A correct alternative in a multiple-choice exam can be viewed as being a signal embedded in noise (incorrect alternatives). Examinees are assumed to have perceptions of the plausibility of each alternative, and the decision process is to choose the most plausible alternative. It is also assumed that each examinee either knows or does not know each item. These assumptions together lead to a signal detection choice model for multiple-choice exams. The model can be viewed, statistically, as a mixture extension, with random mixing, of the traditional choice model, or similarly, as a grade-of-membership extension. A version of the model with extreme value distributions is developed, in which case the model simplifies to a mixture multinomial logit model with random mixing. The approach is shown to offer measures of item discrimination and difficulty, along with information about the relative plausibility of each of the alternatives. The model, parameters, and measures derived from the parameters are compared to those obtained with several commonly used item response theory models. An application of the model to an educational data set is presented.

[1]  Elena A. Erosheva,et al.  Grade of membership and latent structure models with application to disability survey data , 2002 .

[2]  Michael L. Nering,et al.  Handbook of Polytomous Item Response Theory Models , 2010 .

[3]  David Thissen,et al.  A response model for multiple choice items , 1984 .

[4]  Lawrence T. DeCarlo,et al.  On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models , 2010 .

[5]  L. T. DeCarlo.  Signal detection theory with finite mixture distributions: theoretical developments with applications to recognition memory. , 2002, Psychological review.

[6]  D. McFadden.  Conditional logit analysis of qualitative choice behavior , 1972 .

[7]  R. Philip Chalmers,et al.  mirt: A Multidimensional Item Response Theory Package for the R Environment , 2012 .

[8]  Lawrence T. DeCarlo,et al.  An Item Response Model for True–False Exams Based on Signal Detection Theory , 2020, Applied psychological measurement.

[9]  L. Thurstone.  A law of comparative judgment. , 1994 .

[10]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[11]  J SWETS,et al.  Decision processes in perception. , 1961, Psychological review.

[12]  R. Darrell Bock,et al.  Estimating item parameters and latent ability when responses are scored in two or more nominal categories , 1972 .

[13]  T. Wickens.  Elementary Signal Detection Theory , 2001 .

[14]  A. Rivlin,et al.  Economic Choices , 2001 .

[15]  R. Duncan Luce,et al.  Individual Choice Behavior: A Theoretical Analysis , 1979 .

[16]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[17]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[18]  Wim J. van der Linden,et al.  IRT Parameter Estimation With Response Times as Collateral Information , 2010 .

[19]  J. Atchison,et al.  Logistic-normal distributions:Some properties and uses , 1980 .

[20]  H. Akaike.  A new look at the statistical model identification , 1974 .

[21]  Anthony C. Davison,et al.  Statistics of Extremes , 2015, International Encyclopedia of Statistical Science.

[22]  L. T. DeCarlo.  Insights from Reparameterized DINA and Beyond , 2019, Handbook of Diagnostic Classification Models.

[23]  Gustav Theodor Fechner,et al.  Elements of psychophysics , 1966 .

[24]  Lawrence T. DeCarlo,et al.  On a signal detection approach to m-alternative forced choice with bias, with maximum likelihood and Bayesian approaches to estimation , 2012 .

[25]  De Ayala,et al.  The Theory and Practice of Item Response Theory , 2008 .

[26]  R. Darrell Bock,et al.  The Nominal Categories Model , 1997 .

[27]  L. T. DeCarlo.  On the Analysis of Fraction Subtraction Data: The DINA Model, Classification, Latent Class Sizes, and the Q-Matrix , 2011 .

[28]  J. Yellott.  The relationship between Luce's Choice Axiom, Thurstone's Theory of Comparative Judgment, and the double exponential distribution , 1977 .