Testing Person Fit in Cognitive Diagnosis

In cognitive diagnosis, the test-taking behavior of some examinees may be idiosyncratic so that their test scores may not reflect their true cognitive abilities as much as that of more typical examinees. Statistical tests are developed to recognize the following: (a) nonmasters of the required attributes who correctly answer the item (spuriously high scores) and (b) masters of the attributes who fail to correctly answer the item (spuriously low scores). For a person, nonzero probability of aberrant behavior is tested as the alternative hypothesis, against normal behavior as the null hypothesis. The two generalized likelihood ratio test statistics used, with the null hypothesis parameter on the boundary of the parameter space in each, have asymptotic distributions of a 50:50 mixture of a chi-square distribution with one degree of freedom and a degenerate distribution that is a constant of 0 under the null hypothesis. Simulation results, primarily based on the DINA model (deterministic inputs, noisy ‘‘AND’’ gate), are used to investigate the following: (a) how accurately the statistical tests identify normal/aberrant behaviors, (b) how the power of the tests depends on the length of the cognitive exam and the degree of the inclination toward aberrance, and (c) how sensitive the tests are to inaccurate estimation of model parameters.

[1]  Michael V. Levine,et al.  Optimal appropriateness measurement , 1988 .

[2]  R. Okafor.  Maximum likelihood estimation from incomplete data , 1987 .

[3]  Yen-Fen Liao.  Investigating the Construct Validity of the Grammar and Vocabulary Section and the Listening Section of the ECCE: Lexico-Grammatical Ability as a Predictor of L2 Listening Ability , 2007 .

[4]  Herbert Hoijtink,et al.  The many null distributions of person fit indices , 1990 .

[5]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[6]  Frederic M. Lord,et al.  Practical Applications of Item Characteristic Curve Theory. , 1977 .

[7]  Klaas Sijtsma,et al.  Methodology Review: Evaluating Person Fit , 2001 .

[8]  John T. Willse,et al.  Defining a Family of Cognitive Diagnosis Models Using Log-Linear Models with Latent Variables , 2009 .

[9]  Benjamin D. Wright,et al.  Solving measurement problems with the Rasch model. , 1977 .

[10]  Fritz Drasgow,et al.  A Decision-Theoretic Approach to the Use of Appropriateness Measurement for Detecting Invalid Test and Scale Scores , 1987 .

[11]  Edward H. Haertel.  Using restricted latent class models to map the skill structure of achievement items , 1989 .

[12]  Louis V. DiBello,et al.  31A Review of Cognitively Diagnostic Assessment and a Summary of Psychometric Models , 2006 .

[13]  K. Liang,et al.  Asymptotic Properties of Maximum Likelihood Estimators and Likelihood Ratio Tests under Nonstandard Conditions , 1987 .

[14]  B. Wright,et al.  Best test design , 1979 .

[15]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[16]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[17]  F. Lord.  PRACTICAL APPLICATIONS OF ITEM CHARACTERISTIC CURVE THEORY , 1977 .

[18]  W. Emons.  Detection and Diagnosis of Person Misfit From Patterns of Summed Polytomous Item Scores , 2009 .

[19]  James E. Purpura,et al.  Assessing Grammar , 2004 .

[20]  Robert J. Mislevy,et al.  TEST THEORY RECONCEIVED , 1994 .

[21]  James E. Purpura,et al.  Assessing Grammar: Assessing Grammar , 2004 .

[22]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[23]  Rob R. Meijer,et al.  The Number of Guttman Errors as a Simple and Powerful Person-Fit Statistic , 1994 .

[24]  Donald B. Rubin,et al.  Measuring the Appropriateness of Multiple-Choice Test Scores , 1979 .

[25]  Fritz Drasgow,et al.  Appropriateness Measurement: Validating Studies and Variable Ability Models , 1983 .

[26]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[27]  Kikumi K. Tatsuoka,et al.  Indices for Detecting Unusual Patterns: Links Between Two General Approaches and Potential Applications , 1983 .

[28]  Rob R. Meijer,et al.  The Influence of the Presence of Deviant Item Score Patterns on the Power of a Person-Fit Statistic , 1994 .

[29]  Christine E. DeMars,et al.  Item Response Theory , 2010, Assessing Measurement Invariance for Applied Research.

[30]  A. Agresti.  Categorical data analysis , 1993 .

[31]  David J. Weiss,et al.  The Person Response Curve: Fit of Individuals to Item Response Theory Models , 1983 .