Assessing Differential Step Functioning in Polytomous Items Using a Common Odds Ratio Estimator.

Many statistics used in the assessment of differential item functioning (DIF) in polytomous items yield a single item-level index of measurement invariance that collapses information across all response options of the polytomous item. Utilizing a single item-level index of DIF can, however, be misleading if the magnitude or direction of the DIF changes across the steps underlying the polytomous response process. A more comprehensive approach to examining measurement invariance in polytomous item formats is to examine invariance at the level of each step of the polytomous item, a framework described in this article as differential step functioning (DSF). This article proposes a nonparametric DSF estimator that is based on the Mantel-Haenszel common odds ratio estimator (Mantel & Haenszel, 1959), which is frequently implemented in the detection of DIF in dichotomous items. A simulation study demonstrated that when the level of DSF varied in magnitude or sign across the steps underlying the polytomous response options, the DSF-based approach typically provided a more powerful and accurate test of measurement invariance than did corresponding item-level DIF estimators.

[1]  Rebecca Zwick,et al.  Evaluating the Magnitude of Differential Item Functioning in Polytomous Items , 1996 .

[2]  E. Muraki.  A Generalized Partial Credit Model: Application of an EM Algorithm , 1992 .

[3]  T. Yanagawa,et al.  Homogeneity Test with a Generalized Mantel-Haenszel Estimator for L2 × K Contingency Tables , 1990 .

[4]  Robert D. Ankenmann,et al.  An Investigation of the Power of the Likelihood Ratio Goodness-of-Fit Statistic in Detecting Differential Item Functioning. , 1999 .

[5]  Neil J. Dorans,et al.  DIF Assessment for Polytomously Scored Items: A Framework for Classification and Evaluation , 1995 .

[6]  Daniel M. Bolt,et al.  A Monte Carlo Comparison of Parametric and Nonparametric Polytomous DIF Detection Methods , 2002 .

[7]  Jeffrey A Douglas,et al.  Item-Bundle DIF Hypothesis Testing: Identifying Suspect Bundles and Assessing Their Differential Functioning , 1996 .

[8]  R. Zwick,et al.  Assessment of Differential Item Functioning for Performance Tasks , 1993 .

[9]  Nathan Mantel,et al.  Chi-square tests with one degree of freedom , 1963 .

[10]  Allan S. Cohen,et al.  Detection of Differential Item Functioning in the Graded Response Model , 1993 .

[11]  Paul W. Holland,et al.  Estimators of the Variance of the Mantel-Haenszel Log-Odds-Ratio Estimate , 1987 .

[12]  James Algina,et al.  Applying the Liu-Agresti Estimator of the Cumulative Common Odds Ratio to DIF Detection in Polytomous Items , 2003 .

[13]  Gideon J. Mellenbergh,et al.  Conceptual Notes on Models for Discrete Polytomous Item Responses , 1995 .

[14]  Keith F Widaman,et al.  Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. , 1993, Psychological bulletin.

[15]  Mark J. Gierl,et al.  Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory Analysis , 2001 .

[16]  William Stout,et al.  A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF , 1993 .

[17]  R. Darrell Bock,et al.  Estimating item parameters and latent ability when responses are scored in two or more nominal categories , 1972 .

[18]  Wen-Chung Wang,et al.  Factors Influencing the Mantel and Generalized Mantel-Haenszel Methods for the Assessment of Differential Item Functioning in Polytomous Items , 2004 .

[19]  Walter W. Hauck,et al.  The Large Sample Variance of the Mantel-Haenszel Estimator of a Common Odds Ratio , 1979 .

[20]  S. Reise,et al.  Parameter Recovery in the Graded Response Model Using MULTILOG , 1990 .

[21]  Klaas Sijtsma,et al.  On measurement properties of continuation ratio models , 2001 .

[22]  Allan S. Cohen,et al.  Detection of Differential Item Functioning Under the Graded Response Model With the Likelihood Ratio Test , 1998 .

[23]  Hua-Hua Chang,et al.  Detecting DIF for Polytomously Scored Items: An Adaptation of the SIBTEST Procedure , 1995 .

[24]  J. S. Roberts,et al.  A General Item Response Theory Model for Unfolding Unidimensional Polytomous Responses , 2000 .

[25]  Timothy R. Miller,et al.  Logistic Regression and Its Use in Detecting Differential Item Functioning in Polytomous Items , 1996 .

[26]  David Thissen,et al.  A response model for multiple choice items , 1984 .

[27]  D. Bolt.  A SIBTEST Approach to Testing DIF Hypotheses Using Experimentally Designed Test Items , 2000 .

[28]  Neil J. Dorans,et al.  Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. , 1986 .

[29]  Tony C. M. Lam,et al.  Assessing Differential Item Functioning in Performance Assessment: Review and Recommendations. , 2005 .

[30]  Gregory Camilli,et al.  Application of a Method of Estimating DIF for Polytomous Test Items , 1999 .

[31]  Nambury S. Raju,et al.  A Description and Demonstration of the Polytomous-DFIT Framework , 1999 .

[32]  R. Zwick,et al.  ASSESSING DIFFERENTIAL ITEM FUNCTIONING IN PERFORMANCE TESTS , 1993 .

[33]  W. Haenszel,et al.  Statistical aspects of the analysis of data from retrospective studies of disease. , 1959, Journal of the National Cancer Institute.

[34]  Randall D. Penfield.  Applying the Breslow-Day Test of Trend in Odds Ratio Heterogeneity to the Analysis of Nonuniform DIF , 2003 .

[35]  D. Cox.  The Regression Analysis of Binary Sequences , 1958 .

[36]  A Agresti,et al.  Mantel-Haenszel-type inference for cumulative odds ratios with a stratified ordinal response. , 1996, Biometrics.