A Comparison of Item-Fit Statistics for the Three-Parameter Logistic Model

In this article, the Type I error rate and the power of a number of existing and new tests of fit to the 3-parameter logistic model (3PLM) are investigated. The first test is a generalization of a test for the evaluation of the fit to the 2-parameter logistic model (2PLM) based on the Lagrange multiplier (LM) test or the equivalent efficient score test. This technique is applied to two model violations: deviation from the 3PLM item characteristic curve and violation of local stochastic independence. The LM test for the first violation is compared with the Q 1 – G² j and S – G² j tests, respectively. The LM test for the second violation is compared with the Q 3 test and a new test, the S 3 test, which can be viewed as a generalization of the approach of the S – G² j test to the evaluation of violation of local independence. The results of simulation studies indicate that all tests, except the Q 1 – G² j test, have a Type I error rate that is acceptably close to the nominal significance level, and good power to detect the model violations they are targeted at. When, however, misfitting items are present in a test, the proportion of items that are flagged incorrectly as misfitting can become undesirably high, especially for short tests.

[1]  Wendy M. Yen,et al.  Scaling Performance Assessments: Strategies for Managing Local Item Dependence , 1993 .

[2]  Ivo W. Molenaar,et al.  Loglinear Rasch Model Tests , 2005 .

[3]  Calyampudi R. Rao.  Large sample tests of statistical hypotheses concerning several parameters with applications to problems of estimation , 1948, Mathematical Proceedings of the Cambridge Philosophical Society.

[4]  Wendy M. Yen,et al.  Effects of Local Item Dependence on the Fit and Equating Performance of the Three-Parameter Logistic Model , 1984 .

[5]  Cees A. W. Glas,et al.  Statistical tests for differential test functioning in Rasch's model for speed tests , 2001 .

[6]  T. Louis.  Finding the Observed Information Matrix When Using the EM Algorithm , 1982 .

[7]  W. M. Yen.  Using Simulation Results to Choose a Latent Trait Model , 1981 .

[8]  Cees A. W. Glas,et al.  DETECTION OF DIFFERENTIAL ITEM FUNCTIONING USING LAGRANGE MULTIPLIER TESTS , 1996 .

[9]  Noel A Cressie,et al.  Characterizing the manifest probabilities of latent trait models , 1983 .

[10]  Cees A. W. Glas,et al.  The derivation of some tests for the rasch model from the multinomial distribution , 1988 .

[11]  Robert J. Mislevy,et al.  BILOG 3 : item analysis and test scoring with binary logistic models , 1990 .

[12]  N. D. Verhelst,et al.  Extensions of the partial credit model , 1989 .

[13]  Robert J. Mislevy,et al.  Bayes modal estimation in item response models , 1986 .

[14]  Craig N. Mills,et al.  A Comparison of Several Goodness-of-Fit Statistics , 1985 .

[15]  D. Bolt.  Essays on Item Response Theory. A. Boomsma, M. A. J. van Duijn, and T. A. B. Snijders (Eds.) [Book Review]. , 2003 .

[16]  Cornelis A.W. Glas,et al.  Modification indices for the 2-PL and the nominal response model , 1999 .

[17]  Cornelis A.W. Glas,et al.  Differential Item Functioning Depending on General Covariates , 2001 .

[18]  J. Aitchison,et al.  Maximum-Likelihood Estimation of Parameters Subject to Restraints , 1958 .

[19]  Frederic M. Lord,et al.  Comparison of IRT True-Score and Equipercentile Observed-Score "Equatings" , 1984 .

[20]  D. Thissen,et al.  Likelihood-Based Item-Fit Indices for Dichotomous Item Response Theory Models , 2000 .

[21]  Robert J. Jannarone,et al.  Conjunctive item response theory kernels , 1986 .

[22]  Arnold L. van den Wollenberg,et al.  Two new test statistics for the rasch model , 1982 .

[23]  I. W. Molenaar,et al.  Rasch models: foundations, recent developments and applications , 1995 .

[24]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[25]  Anne Boomsma,et al.  Essays on Item Response Theory , 2000 .

[26]  Cees A. W. Glas,et al.  Testing the Rasch Model , 1995 .

[27]  Stephen E. Fienberg,et al.  Discrete Multivariate Analysis: Theory and Practice , 1976 .

[28]  R. D. Bock,et al.  Adaptive EAP Estimation of Ability in a Microcomputer Environment , 1982 .