A new method for detecting differential item functioning in the Rasch model

Differential item functioning (DIF) can lead to an unfair advantage or disadvantage for certain subgroups in educational and psychological testing. Therefore, a variety of statistical methods has been suggested for detecting DIF in the Rasch model. Most of these methods are designed for the comparison of pre-specified focal and reference groups, such as males and females. Latent class approaches, on the other hand, allow to detect previously unknown groups exhibiting DIF. However, this approach provides no straightforward interpretation of the groups with respect to person characteristics. Here we propose a new method for DIF detection based on model-based recursive partitioning that can be considered as a compromise between those two extremes. With this approach it is possible to detect groups of subjects exhibiting DIF, which are not prespecified, but result from combinations of observed ovariates. These groups are directly interpretable and can thus help understand the psychological sources of DIF. The statistical background and construction of the new method is first introduced by means of an instructive example, and then applied to data from a general knowledge quiz and a teaching evaluation.

[1]  E. B. Andersen,et al.  A goodness of fit test for the rasch model , 1973 .

[2]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[3]  Stuart Parkes,et al.  Allgemeinbildung in Deutschland: Erkenntnisse aus dem SPIEGEL-Studentenpisa-Test , 2012 .

[4]  Engelbert Theurl,et al.  Socioeconomic Environment and Mortality: A two-level Decomposition by Sex and Cause of Death , 2010 .

[5]  Michael Kirchler,et al.  Trading strategies and trading profits in experimental asset markets with cumulative information , 2010 .

[6]  Matthias Sutter,et al.  Social preferences during childhood and the role of gender and age -- An experiment in Austria and Sweden , 2011 .

[7]  Achim Zeileis,et al.  Accounting for Individual Differences in Bradley-Terry Models by Means of Recursive Partitioning , 2011 .

[8]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[9]  Gershon Ben-Shakhar,et al.  Gender Differences in Multiple‐Choice Tests: The Role of Differential Guessing Tendencies , 1991 .

[10]  K. Hornik,et al.  Model-Based Recursive Partitioning , 2008 .

[11]  P. Raschky,et al.  Uncertainty of Governmental Relief and the Crowding out of Flood Insurance , 2013 .

[12]  Annette M. Maij-de Meij,et al.  Fitting a Mixture Item Response Theory Model to Personality Questionnaire Data: Characterizing Latent Classes and Investigating Possibilities for Improving Prediction , 2008 .

[13]  M. Sutter,et al.  Working Papers in Economics and Statistics Guilt from Promise-breaking and Trust in Markets for Expert Services -theory and Experiment Guilt from Promise-breaking and Trust in Markets for Expert Services – Theory and Experiment * , 2022 .

[14]  R. Petersen,et al.  Differential item functioning of the Boston Naming Test in cognitively normal African American and Caucasian older adults , 2009, Journal of the International Neuropsychological Society.

[15]  K. Hornik,et al.  Unbiased Recursive Partitioning: A Conditional Inference Framework , 2006 .

[16]  Matthias Sutter,et al.  Equality, Equity and Incentives: An Experiment , 2013, SSRN Electronic Journal.

[17]  D. Andrews Tests for Parameter Instability and Structural Change with Unknown Change Point , 1993 .

[18]  K. Hornik,et al.  Generalized M‐fluctuation tests for parameter instability , 2007 .

[19]  Matthias Sutter,et al.  Psychological Pressure in Competitive Environments: New Evidence from Randomized Natural Experiments , 2012, Manag. Sci..

[20]  Wim Van Den Noortgate,et al.  Assessing and Explaining Differential Item Functioning Using Logistic Mixed Models , 2005 .

[21]  Stefan Lang,et al.  Working Papers in Economics and Statistics Modeling House Prices Using Multilevel Structured Additive Regression Modeling House Prices Using Multilevel Structured Additive Regression , 2022 .

[22]  M. Gächter,et al.  Convergence of the Health Status at the Local Level: Empirical Evidence from Austria , 2010 .

[23]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[24]  M. Gächter,et al.  Retaining the thin blue line: What shapes workers' intentions not to quit the current work environment , 2013 .

[25]  I. W. Molenaar,et al.  Rasch models: foundations, recent developments and applications , 1995 .

[26]  Eric Turkheimer,et al.  Illustration of MIMIC-Model DIF Testing with the Schedule for Nonadaptive and Adaptive Personality , 2009, Journal of psychopathology and behavioral assessment.

[27]  Carolin Strobl,et al.  Wissen Frauen weniger oder nur das Falsche? Ein statistisches Modell für unterschiedliche Aufgaben-Schwierigkeiten in Teilstichproben , 2010 .

[28]  Gregory R. Hancock,et al.  Advances in Latent Variable Mixture Models , 2007 .

[29]  Y.-S. Shih,et al.  A note on split selection bias in classification trees , 2004, Comput. Stat. Data Anal..

[30]  B. Carleton,et al.  The Dimensionality and Gender Differential Item Functioning of the Mini Asthma Quality of Life Questionnaire (MiniAQLQ) , 2004 .

[31]  Matthias Sutter,et al.  Gender, Competition and the Efficiency of Policy Interventions , 2010, SSRN Electronic Journal.

[32]  Henk Kelderman,et al.  Examining differential item functioning due to item difficulty and alternative attractiveness , 1992 .

[33]  Johannes Gehrke,et al.  Bias Correction in Classification Tree Construction , 2001, ICML.

[34]  Allan S. Cohen,et al.  A Mixture Model Analysis of Differential Item Functioning , 2005 .

[35]  G. Tutz,et al.  An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. , 2009, Psychological methods.

[36]  G. Masters A rasch model for partial credit scoring , 1982 .

[37]  W. Stout Psychometrics: From practice to theory and back , 2002 .

[38]  Ruth M. Pfeiffer,et al.  Working Papers in Economics and Statistics Comparing Penalized Splines and Fractional Polynomials for Flexible Modelling of the Effects of Continuous Predictor Variables , 2010 .

[39]  David Huffman,et al.  University of Innsbruck Working Papers in Economics and Statistics Group Membership , Competition , and Altruistic versus Antisocial Punishment : Evidence from Randomly Assigned Army Groups , 2010 .

[40]  M. Hanke,et al.  Football championships and jersey sponsors’ stock prices: an empirical investigation , 2013 .

[41]  G. H. Fischer,et al.  The linear logistic test model as an instrument in educational research , 1973 .

[42]  C. McHorney,et al.  Assessment of Differential Item Functioning for Demographic Comparisons in the MOS SF-36 Health Survey , 2006, Quality of Life Research.

[43]  Achim Zeileis,et al.  psychotree - Recursive partitioning based on psychometric models: Version 0.12-1 , 2011 .

[44]  M. Liou More on the Computation of Higher-Order Derivatives of the Elementary Symmetric Functions in the Rasch Model , 1994 .

[45]  Carolin Strobl,et al.  Unbiased split selection for classification trees based on the Gini Index , 2007, Comput. Stat. Data Anal..

[46]  Stefan Lang,et al.  Working Papers in Economics and Statistics Applications of Multilevel Structured Additive Regression Models to Insurance Data Applications of Multilevel Structured Additive Regression Models to Insurance Data , 2022 .

[47]  Jürgen Rost,et al.  Rasch Models in Latent Classes: An Integration of Two Approaches to Item Analysis , 1990 .

[48]  Francis Tuerlinckx,et al.  A nonlinear mixed model framework for item response theory. , 2003, Psychological methods.

[49]  P. Mair,et al.  Extended Rasch Modeling: The eRm Package for the Application of IRT Models in R , 2007 .

[50]  M. Sutter,et al.  Strategic Sophistication of Individuals and Teams in Experimental Normal-Form Games , 2010, SSRN Electronic Journal.

[51]  Matthias Sutter,et al.  University of Innsbruck Working Papers in Economics and Statistics Household Decision Making in Rural China : Using Experiments to Estimate the Influences of Spouses , 2010 .

[52]  Matthias Sutter,et al.  Teams Make You Smarter: Learning and Knowledge Transfer in Auctions and Markets by Teams and Individuals , 2010, SSRN Electronic Journal.

[53]  Jesús Crespo-Cuaresma,et al.  Business cycle convergence in EMU : A first look at the second moment * , 2010 .