A New Online Calibration Method for Multidimensional Computerized Adaptive Testing

Multidimensional-Method A (M-Method A) has been proposed as an efficient and effective online calibration method for multidimensional computerized adaptive testing (MCAT) (Chen & Xin, Paper presented at the 78th Meeting of the Psychometric Society, Arnhem, The Netherlands, 2013). However, a key assumption of M-Method A is that it treats person parameter estimates as their true values, thus this method might yield erroneous item calibration when person parameter estimates contain non-ignorable measurement errors. To improve the performance of M-Method A, this paper proposes a new MCAT online calibration method, namely, the full functional MLE-M-Method A (FFMLE-M-Method A). This new method combines the full functional MLE (Jones & Jin in Psychometrika 59:59–75, 1994; Stefanski & Carroll in Annals of Statistics 13:1335–1351, 1985) with the original M-Method A in an effort to correct for the estimation error of ability vector that might otherwise adversely affect the precision of item calibration. Two correction schemes are also proposed when implementing the new method. A simulation study was conducted to show that the new method generated more accurate item parameter estimation than the original M-Method A in almost all conditions.

[1]  Daniel O. Segall,et al.  General ability measurement: An application of multidimensional item response theory , 2001 .

[2]  W. D. Linden,et al.  Optimal Bayesian Adaptive Design for Test-Item Calibration , 2015, Psychometrika.

[3]  D. Hall.  Measurement Error in Nonlinear Models: A Modern Perspective , 2008 .

[4]  N. Verhelst,et al.  Item calibration in incomplete testing designs , 2011 .

[5]  Jae-Chun Ban,et al.  Data Sparseness and Online Pretest Item Calibration/Scaling Methods in CAT. ACT Research Report Series. , 2002 .

[6]  Daniel O. Segall,et al.  Multidimensional adaptive testing , 1996 .

[7]  Chun Wang,et al.  Improving Measurement Precision of Hierarchical Latent Traits Using Adaptive Testing , 2014 .

[8]  Gerard T. Barkema,et al.  Monte Carlo Methods in Statistical Physics , 1999 .

[9]  Lihua Yao,et al.  Using Multidimensional CAT to Administer a Short, Yet Precise, Screening Test , 2014 .

[10]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[11]  Chun Wang,et al.  On Latent Trait Estimation in Multidimensional Compensatory Item Response Models , 2015, Psychometrika.

[12]  Ke-Hai Yuan,et al.  The Impact of Fallible Item Parameter Estimates on Latent Trait Recovery , 2010, Psychometrika.

[13]  Robert J. Mislevy,et al.  Does adaptive testing violate local independence? , 2000 .

[14]  Sebastian Weirich,et al.  Modeling Booklet Effects for Nonequivalent Group Designs in Large-Scale Assessment , 2015, Educational and psychological measurement.

[15]  Hua-Hua Chang,et al.  Deriving Stopping Rules for Multidimensional Computerized Adaptive Testing , 2013 .

[16]  Wim J. van der Linden,et al.  Multidimensional Adaptive Testing with Optimal Design Criteria for Item Selection , 2008, Psychometrika.

[17]  Robert J. Mislevy,et al.  Bayes modal estimation in item response models , 1986 .

[18]  Hua-Hua Chang,et al.  Item Selection in Multidimensional Computerized Adaptive Testing—Gaining Information from Different Angles , 2011 .

[19]  Shelby J. Haberman,et al.  LINKING PARAMETER ESTIMATES DERIVED FROM AN ITEM RESPONSE MODEL THROUGH SEPARATE CALIBRATIONS , 2009 .

[20]  Representation of Competencies in Multidimensional IRT Models with Within-Item and Between-Item Multidimensionality , 2008 .

[21]  Lihua Yao.  Comparing the Performance of Five Multidimensional CAT Selection Procedures With Different Stopping Rules , 2013 .

[22]  Johannes Hartig,et al.  Student, School, and Country Differences in Sustained Test-Taking Effort in the 2009 PISA Reading Assessment , 2014 .

[23]  Neil J. Dorans,et al.  Item Response Theory, Item Calibration, and Proficiency Estimation , 2000 .

[24]  Frederic M. Lord.  Tailored Testing, an Application of Stochastic Approximation , 1971 .

[25]  Jae-Chun Ban,et al.  A Comparative Study of On‐line Pretest Item—Calibration/Scaling Methods in Computerized Adaptive Testing , 2001 .

[26]  Rianne Janssen,et al.  Modeling Item-Position Effects Within an IRT Framework , 2012 .

[27]  Zhiying Jin,et al.  Optimal sequential designs for on-line item estimation , 1994 .

[28]  Hua-Hua Chang,et al.  Kullback–Leibler Information and Its Applications in Multi-Dimensional Adaptive Testing , 2011 .

[29]  Raymond J. Carroll,et al.  Covariate Measurement Error in Logistic Regression , 1985 .

[30]  Hua-Hua Chang,et al.  The asymptotic posterior normality of the latent trait in an IRT model , 1993 .

[31]  M. Reckase.  Multidimensional Item Response Theory , 2009 .

[32]  Raymond J. Adams,et al.  The Multidimensional Random Coefficients Multinomial Logit Model , 1997 .

[33]  D. Lien.  Moments of truncated bivariate log-normal distributions☆ , 1985 .

[34]  Hua-Hua Chang,et al.  Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT , 2012 .

[35]  Frank B. Baker,et al.  Item Response Theory : Parameter Estimation Techniques, Second Edition , 2004 .