Implications of the Multidimensionality-Based DIF Analysis Framework for Selecting a Matching and Studied Subtest

In this paper we describe and illustrate the Roussos-Stout (1996) multidimensionality-based DIF analysis framework, with emphasis on its implication for the selection of a matching and studied subtest for DIF analyses. Standard DIF practice encourages an exploratory search for matching subtest items based on purely statistical criteria, such as a failure to display DIF. By contrast, the multidimensional DIF framework emphasizes a substantively-informed selection of items for both the matching and studied subtest based on the dimensions suspected of underlying the test data. Using two examples, we demonstrate that these two approaches lead to different interpretations about the occurrence of DIF in a test. It is argued that selecting a valid matching and studied subtest, as implied by the multidimensional framework, can lead to a more informed understanding of why DIF occurs. Bias occurs when tests yield scores or promote score interpretations that result in different meanings for members of different groups. Bias is often attributed to construct-irrelevant dimensions that differentially affect the test scores for different groups of examinees (Standards for Educational and Psychological Testing, 1999). Group differences can also be attributed to item impact. Impact occurs when construct-relevant dimensions differentially affect the tests scores for different groups of examinees. In this case, the item is a relevant measure of the target construct and the difference between the groups reflects a true difference on that construct. Differential item functioning (DIF) studies are designed to identify and interpret these construct-related dimensions using a combination of statistical and substantive analyses. The statistical analysis involves administering the test, matching members of the reference and focal group on a measure of ability derived from that test, and using statistical procedures to identify group differences on test items. An item exhibits DIF when examinees from the reference and focal groups differ in the probability of answering that item correctly, after controlling for ability. The substantive analysis builds on the statistical analysis because DIF items are often scrutinized by expert reviewers (e.g., test developers or content specialists) who attempt to identify the construct-related dimensions that produce group differences. A DIF item is considered biased when reviewers identify some dimension, deemed to be irrelevant to the construct measured by the test, that places one group of examinees at a disadvantage. Conversely, a DIF item displays impact when the dimension that differentiates the groups is judged to be relevant to the construct measured by the test. Considerable …