An empirical investigation demonstrating the multidimensional DIF paradigm: A cognitive explanation for DIF

Differential Item Functioning (DIF) is traditionally used to identify different item performance patterns between intact groups, most commonly involving race or sex comparisons. This study advocates expanding the utility of DIF as a step in construct validation. Rather than grouping examinees based on cultural differences, the reference and focal groups are chosen from two extremes along a distinct cognitive dimension that is hypothesized to supplement the dominant latent trait being measured. Specifically, this study investigates DIF between proficient and non-proficient fourth- and seventh-grade writers on open-ended mathematics test items that require students to communicate about mathematics. It is suggested that the occurrence of DIF in this situation actually enhances, rather than detracts from, the construct validity of the test because, according to the National Council of Teachers of Mathematics (NCTM), mathematical communication is an important component of mathematical ability, the dominant construct being assessed. However, the presence of DIF influences the validity of inferences that can be made from test scores and suggests that two scores should be reported, one for general mathematical ability and one for mathematical communication. The fact that currently only one test score is reported, a simple composite of scores on multiple-choice and open-ended items, may lead to incorrect decisions being made about examinees.

[1]  C. Hirsch.  Curriculum and Evaluation Standards for School Mathematics , 1988 .

[2]  R. Hambleton,et al.  Assessing the Dimensionality of a Set of Test Items , 1986 .

[3]  Terry A. Ackerman,et al.  An Examination of Conditioning Variables Used in Computer Adaptive Testing for DIF Analyses , 2001 .

[4]  J. Scheuneman,et al.  Using Differential Item Functioning Procedures to Explore Sources of Item Difficulty and Group Performance Characteristics. , 1990 .

[5]  Jeffrey A Douglas,et al.  Item-Bundle DIF Hypothesis Testing: Identifying Suspect Bundles and Assessing Their Differential Functioning , 1996 .

[6]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[7]  Terry A. Ackerman.  A Didactic Explanation of Item Bias, Item Impact, and Item Validity from a Multidimensional Perspective , 1992 .

[8]  William Stout,et al.  Using New Proximity Measures With Hierarchical Cluster Analysis to Detect Multidimensionality , 1998 .

[9]  Ratna Nandakumar,et al.  Simultaneous DIF Amplification and Cancellation: Shealy-Stout's Test for DIF , 1993 .

[10]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[11]  William Stout,et al.  The theoretical detect index of dimensionality and its application to approximate simple structure , 1999 .

[12]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[13]  William Stout,et al.  A Multidimensionality-Based DIF Analysis Paradigm , 1996 .

[14]  S. Lane,et al.  Detection of Gender-Related Differential Item Functioning in a Mathematics Performance Assessment. , 1996 .

[15]  W. H. Angoff,et al.  Perspectives on differential item functioning methodology. , 1993 .

[16]  K. Jöreskog,et al.  LISREL 8: New Statistical Features , 1999 .

[17]  Brian Habing,et al.  Conditional Covariance-Based Nonparametric Multidimensionality Assessment , 1996 .

[18]  Robert J. Mislevy,et al.  Recent Developments in the Factor Analysis of Categorical Variables , 1986 .

[19]  D. Knol,et al.  Empirical Comparison Between Factor Analysis and Multidimensional Item Response Models. , 1991, Multivariate behavioral research.

[20]  Terry A. Ackerman,et al.  The Influence of Conditioning Scores In Performing DIF Analyses , 1994 .

[21]  Terry A. Ackerman.  Creating a Test Information Profile for a Two-Dimensional Latent Space , 1994 .

[22]  A. Su,et al.  The National Council of Teachers of Mathematics , 1932, The Mathematical Gazette.

[23]  Ratna Nandakumar,et al.  Assessing Dimensionality of a Set of Items - Comparison of Different Approaches , 1992 .

[24]  S. Beretvas,et al.  Using Multidimensional versus Unidimensional Ability Estimates To Determine Student Proficiency in Mathematics. , 2000 .

[25]  John Hattie,et al.  An Empirical Study of Various Indices for Determining Unidimensionality. , 1984, Multivariate behavioral research.

[26]  M. Hertzog,et al.  The Assessment of Dimensionality for Use in Item Response Theory. , 1991, Multivariate behavioral research.

[27]  William Stout,et al.  A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF , 1993 .

[28]  Alieia P. Sehmitt,et al.  Differential Item Functioning for Minority Examinees on the SAT , 1990 .

[29]  R. Nandakumar.  Assessing Dimensionality of a Set of Item Responses--Comparison of Different Approaches. , 1994 .

[30]  Hua-Hua Chang,et al.  Detecting DIF for Polytomously Scored Items: An Adaptation of the SIBTEST Procedure , 1995 .

[31]  H. Wainer,et al.  Differential Item Functioning. , 1994 .

[32]  Ulf Olsson,et al.  Maximum likelihood estimation of the polychoric correlation coefficient , 1979 .