Trait Parameter Recovery Using Multidimensional Computerized Adaptive Testing in Reading and Mathematics

Under a multidimensional item response theory (MIRT) computerized adaptive testing (CAT) testing scenario, a trait estimate (θ) in one dimension will provide clues for subsequently seeking a solution in other dimensions. This feature may enhance the efficiency of MIRT CAT’s item selection and its scoring algorithms compared with its counterpart, the unidimensional CAT (UCAT). The present study used existing Reading and Math test data to generate simulated item parameters. A confirmatory item factor analysis model was applied to the data using NOHARM to produce interpretable MIRT item parameters. Results showed that MIRT CAT, conditional on the constraints, was quite capable of producing accurate estimates on both measures. Compared with UCAT, MIRT CAT slightly increased the accuracy of both trait estimates, especially for the low-level or high-level trait examinees in both measures, and reduced the rate of unused items in the item pool.

[1]  T. Theunissen.  Some Applications of Optimization Algorithms in Test Design and Adaptive Testing , 1986 .

[2]  D. Segall.  Principles of Multidimensional Adaptive Testing , 2000 .

[3]  Bernard P. Veldkamp,et al.  Multidimensional adaptive testing with constraints on test content , 2002 .

[4]  Wim J. van der Linden,et al.  A Model for Optimal Constrained Adaptive Testing , 1998 .

[5]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[6]  Hua-Hua Chang,et al.  A Global Information Approach to Computerized Adaptive Testing , 1996 .

[7]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[8]  Willem J. van der Linden,et al.  Multidimensional Adaptive Testing with a Minimum Error-Variance Criterion , 1999 .

[9]  Donald Hedeker,et al.  Full-information item bi-factor analysis , 1992 .

[10]  Richard M. Luecht,et al.  Multidimensional Computerized Adaptive Testing in a Certification or Licensure Context , 1996 .

[11]  R. Lissitz,et al.  An Evaluation of the Accuracy of Multidimensional IRT Linking , 2000 .

[12]  Seock-Ho Kim.  BILOG 3 for Windows: Item Analysis and Test Scoring with Binary Logistic Models , 1997 .

[13]  Mark D. Reckase,et al.  Item Response Theory: Parameter Estimation Techniques , 1998 .

[14]  T. Theunissen.  Binary programming and test design , 1985 .

[15]  Vicente Ponsoda,et al.  A Comparison of Item Exposure Control Methods in Computerized Adaptive Testing , 1998 .

[16]  Mark D. Reckase,et al.  That Measure More Than One Ability , 1985 .

[17]  D. Thissen,et al.  Factor analysis for items scored in two categories , 2000 .

[18]  van der Linden,et al.  Constrained adaptive testing with shadow tests , 2000 .

[19]  Terry A. Ackerman.  A Didactic Explanation of Item Bias, Item Impact, and Item Validity from a Multidimensional Perspective , 1992 .

[20]  Terry A. Ackerman.  Using multidimensional item response theory to understand what items and tests are measuring , 1994 .

[21]  R. Owen,et al.  A Bayesian Sequential Procedure for Quantal Response in the Context of Adaptive Mental Testing , 1975 .

[22]  W. D. Schafer,et al.  Increasing the Homogeneity of CAT's Item‐Exposure Rates by Minimizing or Maximizing Varied Target Functions While Assembling Shadow Tests , 2005 .

[23]  Daniel O. Segall,et al.  General ability measurement: An application of multidimensional item response theory , 2001 .

[24]  M. F. Levine.  COPI " Navy Personnel Research and Development Center , 2022 .

[25]  Willem J. van der Linden,et al.  Optimal Assembly of Psychological and Educational Tests , 1998 .

[26]  W. D. Linden,et al.  A maximin model for IRT-based test design with practical constraints , 1989 .

[27]  Item Banks.  A Maximin Model for Test Design with Practical Constraints , 2007 .

[28]  Robert J. Mislevy,et al.  BILOG 3 : item analysis and test scoring with binary logistic models , 1990 .

[29]  Mark D. Reckase,et al.  A Linear Logistic Multidimensional Model for Dichotomous Item Response Data , 1997 .

[30]  Daniel O. Segall,et al.  Multidimensional adaptive testing , 1996 .

[31]  Mark D. Reckase,et al.  The Difficulty of Test Items That Measure More Than One Ability , 1985 .

[32]  Michael J. Mazu,et al.  Sampling Methodologies With Applications , 2002, Technometrics.

[33]  R. Hambleton,et al.  Handbook of Modern Item Response Theory , 1997 .

[34]  Colin Fraser,et al.  NOHARM: Least Squares Item Factor Analysis. , 1988, Multivariate behavioral research.