Application of a Multidimensional Nested Logit Model to Multiple-Choice Test Items

Nested logit models have been presented as an alternative to multinomial logistic models for multiple-choice test items (Suh and Bolt in Psychometrika 75:454–473, 2010) and possess a mathematical structure that naturally lends itself to evaluating the incremental information provided by attending to distractor selection in scoring. One potential concern in attending to distractors is the possibility that distractor selection reflects a different trait/ability than that underlying the correct response. This paper illustrates a multidimensional extension of a nested logit item response model that can be used to evaluate such distinctions and also defines a new framework for incorporating collateral information from distractor selection when differences exist. The approach is demonstrated in application to questions faced by a university testing center over whether to incorporate distractor selection into the scoring of its multiple-choice tests. Several empirical examples are presented.

[1]  F. Samejima.  A General Model for Free Response Data. , 1972 .

[2]  R. Hambleton,et al.  Item Response Theory , 1984, The History of Educational Measurement.

[3]  Richard J. Patz,et al.  A Straightforward Approach to Markov Chain Monte Carlo Methods for Item Response Models , 1999 .

[4]  Michael C. Rodriguez.  Construct Equivalence of Multiple-Choice and Constructed-Response Items: A Random Effects Synthesis of Correlations , 2003 .

[5]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[6]  Frank B. Baker,et al.  Item Response Theory : Parameter Estimation Techniques, Second Edition , 2004 .

[7]  Daniel M. Bolt,et al.  A Comparison of Alternative Models for Testlets , 2006 .

[8]  David Thissen,et al.  A response model for multiple choice items , 1984 .

[9]  Ernst Wit,et al.  Borrowing strength: a likelihood ratio test for related sparse signals , 2012, Bioinform..

[10]  S. Geisser,et al.  A Predictive Approach to Model Selection , 1979 .

[11]  Klaas Sijtsma,et al.  New Developments in Categorical Data Analysis for the Social and Behavioral Sciences , 2005 .

[12]  Wim J. van der Linden,et al.  IRT Parameter Estimation With Response Times as Collateral Information , 2010 .

[13]  R. Darrell Bock,et al.  Estimating item parameters and latent ability when responses are scored in two or more nominal categories , 1972 .

[14]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[15]  Gunter Maris,et al.  The Nedelsky model for multiple-choice items. , 2005 .

[16]  James A. Wollack,et al.  Recovery of Item Parameters in the Nominal Response Model: A Comparison of Marginal Maximum Likelihood Estimation and Markov Chain Monte Carlo Estimation , 2002 .

[17]  Richard J. Patz,et al.  Making the Most of What We Have: A Practical Application of Multidimensional Item Response Theory in Test Scoring , 2005 .

[18]  D. Borsboom.  Educational Measurement (4th ed.) , 2009 .

[19]  Daniel M. Bolt,et al.  Addressing Score Bias and Differential Item Functioning Due to Individual Differences in Response Style , 2009 .

[20]  C. Robert,et al.  Bayesian Modeling Using WinBUGS , 2009 .

[21]  Li Cai,et al.  The Nominal Categories Item Response Model , 2010 .

[22]  A. Raftery,et al.  How Many Iterations in the Gibbs Sampler , 1991 .

[23]  Jimmy de la Torre,et al.  Improving the Quality of Ability Estimates Through Multidimensional Scoring and Incorporation of Ancillary Variables , 2009 .

[24]  D. Bolt,et al.  Nested Logit Models for Multiple-Choice Item Response Data , 2010 .

[25]  F. Baker,et al.  Item response theory : parameter estimation techniques , 1993 .

[26]  David Thissen,et al.  A taxonomy of item response models , 1986 .

[27]  G. Tutz.  Sequential item response models with an ordered response , 1990 .

[28]  A. Gelfand,et al.  Bayesian Model Choice: Asymptotics and Exact Calculations , 1994 .