Numerical Differentiation Methods for Computing Error Covariance Matrices in Item Response Theory Modeling

In item response theory (IRT) modeling, the item parameter error covariance matrix plays a critical role define? in statistical inference procedures. When item parameters are estimated using the EM algorithm, the parameter error covariance matrix is not an automatic by-product of item calibration. Cai proposed the use of Supplemented EM algorithm for computing the item parameter error covariance matrix. This method has been subsequently implemented in commercial IRT software programs such as IRTPRO and flexMIRT. Jamshidian and Jennrich noted that Supplemented EM is among a class of methods based on numerically differentiating the EM map, and they proposed noniterative alternatives, such as forward difference and Richardson extrapolation, that are mathematically simpler and may lead to a reduction in computational burden when compared with Supplemented EM. However, the relative merits of the various numerical differentiation methods have not been evaluated in the context of IRT modeling. We perform such an evaluation, using both simulated and empirical data. It is found that the accuracy of the simpler noniterative alternatives is heavily dependent on the choice of the numerical differentiation perturbation constants. On the other hand, Supplemented EM consistently maintains accuracy and does not require the selection of perturbation constants. Furthermore, when implemented with an adaptive iteration scheme, an updated Supplemented EM algorithm can be as computationally efficient as the alternatives. The expected (Fisher) information matrix, while accurate, requires too heavy computation for realistic test lengths. Therefore, we recommend the routine use of the updated Supplemented EM algorithm in IRT applications.

[1]  Li Cai,et al.  Characterizing Sources of Uncertainty in Item Response Theory Scale Scores , 2012, Educational and psychological measurement.

[2]  Li Cai,et al.  Generalized full-information item bifactor analysis. , 2011, Psychological methods.

[3]  S. Reise,et al.  Using the Nominal Response Model to Evaluate Response Category Discrimination in the PROMIS Emotional Distress Item Pools , 2011 .

[4]  Li Cai,et al.  A Two-Tier Full-Information Item Factor Analysis Model with Applications , 2010 .

[5]  Li Cai,et al.  The Nominal Categories Item Response Model , 2010 .

[6]  Ke-Hai Yuan,et al.  The Impact of Fallible Item Parameter Estimates on Latent Trait Recovery , 2010, Psychometrika.

[7]  M. Langer.  A reexamination of Lord's Wald test for differential item functioning using item response theory and modern error estimation , 2008 .

[8]  Li Cai,et al.  SEM of another flavour: two new applications of the supplemented EM algorithm. , 2008, The British journal of mathematical and statistical psychology.

[9]  K. Shigemasu,et al.  Standard Errors of Estimated Latent Variable Scores With Estimated Structural Parameters , 2008 .

[10]  D. Thissen,et al.  Limited-information goodness-of-fit testing of item response theory models for sparse 2 tables. , 2006, The British journal of mathematical and statistical psychology.

[11]  Frank B. Baker,et al.  Item Response Theory : Parameter Estimation Techniques, Second Edition , 2004 .

[12]  Mark D. Reckase,et al.  Item Response Theory: Parameter Estimation Techniques , 1998 .

[13]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[14]  Allan S. Cohen,et al.  Detection of Differential Item Functioning in Multiple Groups. , 1995 .

[15]  Ronald K. Hambleton,et al.  Item Parameter Estimation Errors and Their Influence on Test Information Functions , 1994 .

[16]  Robert J. Mislevy,et al.  DEALING WITH UNCERTAINTY ABOUT ITEM PARAMETERS: EXPECTED RESPONSE FUNCTIONS , 1994 .

[17]  Xiao-Li Meng,et al.  Using EM to Obtain Asymptotic Variance-Covariance Matrices: The SEM Algorithm , 1991 .

[18]  R. Tsutakawa,et al.  The effect of uncertainty of item parameter estimation on ability estimates , 1990 .

[19]  I. Meilijson.  A fast improvement to the EM algorithm on its own terms , 1989 .

[20]  R. Darrell Bock,et al.  Item Pool Maintenance in the Presence of Item Parameter Drift. , 1988 .

[21]  David Thissen,et al.  A taxonomy of item response models , 1986 .

[22]  H. Wainer,et al.  Confidence Envelopes for Item Response Theory , 1983 .

[23]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters , 1982 .

[24]  David Thissen,et al.  Marginal maximum likelihood estimation for the one-parameter logistic model , 1982 .

[25]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm , 1981 .

[26]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[27]  M. Kupperman.  Linear Statistical Inference and Its Applications 2nd Edition (C. Radhakrishna Rao) , 1975 .

[28]  R. Darrell Bock,et al.  Estimating item parameters and latent ability when responses are scored in two or more nominal categories , 1972 .

[29]  R. Darrell Bock,et al.  Fitting a response model forn dichotomously scored items , 1970 .

[30]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1968 .

[31]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[32]  Rory A. Fisher,et al.  Theory of Statistical Estimation , 1925, Mathematical Proceedings of the Cambridge Philosophical Society.

[33]  Raymond J. Adams,et al.  PISA 2000 technical report , 2002 .

[34]  R. Jennrich,et al.  Standard errors for EM estimation , 2000 .

[35]  T. Louis.  Finding the Observed Information Matrix When Using the EM Algorithm , 1982 .

[36]  Stephen E. Fienberg,et al.  Discrete Multivariate Analysis: Theory and Practice , 1976 .

[37]  M. Woodbury.  A missing information principle: theory and applications , 1972 .