Model Selection Indices for Polytomous Items

This study examines the utility of four indices for use in model selection with nested and nonnested polytomous item response theory (IRT) models: a cross-validation index and three information-based indices. Four commonly used polytomous IRT models are considered: the graded response model, the generalized partial credit model, the partial credit model, and the rating scale model. In a simulation study, comparisons among the four indices suggest that model selection is dependent to some extent on the particular conditions simulated. Overall, the Bayesian information criterion index appears to be most accurate in selecting the correct polytomous IRT model. Results are presented from analysis of a real data set to illustrate the use of the four indices for selecting an appropriate model.

[1]  Fritz Drasgow,et al.  Distinguishing Among Paranletric item Response Models for Polychotomous Ordered Data , 1994 .

[2]  M. Forster.  Chapter 3: Simplicity and Unification in Model Selection , 2004 .

[3]  R. J. De Ayala,et al.  A Comparison of the Partial Credit and Graded Response Models in Computerized Adaptive Testing , 1992 .

[4]  Terry A. Ackerman.  The Use of Unidimensional Parameter Estimates of Multidimensional Items in Adaptive Testing , 1991 .

[5]  E. Sober.  Instrumentalism, Parsimony, and the Akaike Framework , 2002, Philosophy of Science.

[6]  Bert F. Green,et al.  Adaptive Estimation When the Unidimensionality Assumption of IRT is Violated , 1989 .

[7]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[8]  Remo Ostini,et al.  Polytomous Item Response Theory Models , 2005 .

[9]  Gregory Camilli,et al.  The Effects of Dimensionality on Equating the Law School Admission Test , 1995 .

[10]  D. Bolt.  Evaluating the Effects of Multidimensionality on IRT True-Score Equating , 1999 .

[11]  M. Forster,et al.  Model Selection in Science: The Problem of Language Variance , 1999, The British Journal for the Philosophy of Science.

[12]  Taehoon Kang,et al.  Choosing a Polytomous IRT Model using Bayesian Model Selection Methods , 2006 .

[13]  Howard Wainer,et al.  Estimating Ability With the Wrong Model , 1987 .

[14]  A. Gelfand,et al.  Bayesian Model Choice: Asymptotics and Exact Calculations , 1994 .

[15]  The Effect of Error in Item Parameter Estimates on the Test Response Function Method of Linking , 2001 .

[16]  Klaas Sijtsma,et al.  Detection of Aberrant Item Score Patterns: A Review of Recent Developments. Research Report 94-8. , 1994 .

[17]  P. Fayers.  Item Response Theory for Psychologists , 2004, Quality of Life Research.

[18]  Neil J. Dorans,et al.  THE EFFECTS OF VIOLATIONS OF UNIDIMENSIONALITY ON THE ESTIMATION OF ITEM AND ABILITY PARAMETERS AND ON ITEM RESPONSE THEORY EQUATING OF THE GRE VERBAL SCALE , 1985 .

[19]  Mark A. Pitt,et al.  The Use of MDL to Select among Computational Models of Cognition , 2000, NIPS.

[20]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[21]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1968 .

[22]  Seock-Ho Kim,et al.  A Comparison of Linking and Concurrent Calibration Under the Graded Response Model , 1997 .

[23]  S. Natasha Beretvas,et al.  Comparing Multidimensional and Unidimensional Proficiency Classifications: Multidimensional IRT as a Diagnostic Aid , 2003 .

[24]  H. Akaike.  A new look at the statistical model identification , 1974 .

[25]  David Thissen,et al.  A taxonomy of item response models , 1986 .

[26]  Allan S. Cohen,et al.  IRT Model Selection Methods for Dichotomous Items , 2007 .

[27]  Fritz Drasgow,et al.  Choice of Test Model for Appropriateness Measurement , 1982 .

[28]  E. Muraki.  A GENERALIZED PARTIAL CREDIT MODEL: APPLICATION OF AN EM ALGORITHM , 1992 .

[29]  S. Sahu.  Bayesian Estimation and Model Choice in Item Response Models , 2002 .

[30]  S. Sclove.  Application of model-selection criteria to some problems in multivariate analysis , 1987 .

[31]  P. Boeck,et al.  Confirmatory Analyses of Componential Test Structure Using Multidimensional Item Response Theory. , 1999, Multivariate behavioral research.

[32]  M. Meulders,et al.  A conceptual and psychometric framework for distinguishing categories and dimensions. , 2005, Psychological review.

[33]  G. Masters.  A rasch model for partial credit scoring , 1982 .

[34]  N. G. Best,et al.  WinBUGS User Manual: Version 1.4 , 2001 .

[35]  Model selection in non-nested hidden Markov models for ion channel gating. , 2001, Journal of theoretical biology.

[36]  Brenda H. Loyd,et al.  VERTICAL EQUATING USING THE RASCH MODEL , 1980 .

[37]  C. Mitchell Dayton,et al.  Model Selection Information Criteria for Non-Nested Latent Class Models , 1997 .

[38]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[39]  Christine E. DeMars.  Scoring Subscales Using Multidimensional Item Response Theory Models. , 2005 .

[40]  Allan S. Cohen,et al.  A Mixture Item Response Model for Multiple-Choice Data , 2001 .

[41]  Daniel M. Bolt,et al.  A Monte Carlo Comparison of Parametric and Nonparametric Polytomous DIF Detection Methods , 2002 .

[42]  Christopher Hitchcock,et al.  Prediction Versus Accommodation and the Risk of Overfitting , 2004, The British Journal for the Philosophy of Science.

[43]  Andrew Gelman,et al.  General methods for monitoring convergence of iterative simulations , 1998 .

[44]  D. Andrich.  Application of a Psychometric Rating Model to Ordered Categories Which Are Scored with Successive Integers , 1978 .

[45]  B. Muthén,et al.  Investigating population heterogeneity with factor mixture models. , 2005, Psychological methods.

[46]  Christine E. DeMars.  Sample Size and the Recovery of Nominal Response Model Item Parameters , 2003 .

[47]  Daniel M. Bolt,et al.  Estimation of Compensatory and Noncompensatory Multidimensional Item Response Models Using Markov Chain Monte Carlo , 2003 .

[48]  S. Geisser,et al.  A Predictive Approach to Model Selection , 1979 .

[49]  Allan S. Cohen,et al.  Model Selection Methods for Mixture Dichotomous IRT Models , 2009 .