Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model

A variety of statistical methods have been suggested for detecting differential item functioning (DIF) in the Rasch model. Most of these methods are designed for the comparison of pre-specified focal and reference groups, such as males and females. Latent class approaches, on the other hand, allow the detection of previously unknown groups exhibiting DIF. However, this approach provides no straightforward interpretation of the groups with respect to person characteristics. Here, we propose a new method for DIF detection based on model-based recursive partitioning that can be considered as a compromise between those two extremes. With this approach it is possible to detect groups of subjects exhibiting DIF, which are not pre-specified, but result from combinations of observed covariates. These groups are directly interpretable and can thus help generate hypotheses about the psychological sources of DIF. The statistical background and construction of the new method are introduced by means of an instructive example, and extensive simulation studies are presented to support and illustrate the statistical properties of the method, which is then applied to empirical data from a general knowledge quiz. A software implementation of the method is freely available in the R system for statistical computing.

[1]  M. Sutter,et al.  Working Papers in Economics and Statistics Guilt from Promise-breaking and Trust in Markets for Expert Services -theory and Experiment Guilt from Promise-breaking and Trust in Markets for Expert Services – Theory and Experiment * , 2022 .

[2]  Michael Kirchler,et al.  Trading strategies and trading profits in experimental asset markets with cumulative information , 2010 .

[3]  M. Liou.  More on the Computation of Higher-Order Derivatives of the Elementary Symmetric Functions in the Rasch Model , 1994 .

[4]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[5]  G. Tutz,et al.  An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. , 2009, Psychological methods.

[6]  Allan S. Cohen,et al.  A Mixture Model Analysis of Differential Item Functioning , 2005 .

[7]  Annette M. Maij-de Meij,et al.  Fitting a Mixture Item Response Theory Model to Personality Questionnaire Data: Characterizing Latent Classes and Investigating Possibilities for Improving Prediction , 2008 .

[8]  K. Gabriel,et al.  On closed testing procedures with special reference to ordered analysis of variance , 1976 .

[9]  M. Gächter,et al.  Retaining the thin blue line: What shapes workers' intentions not to quit the current work environment , 2013 .

[10]  Randall D. Penfield.  Assessing Differential Step Functioning in Polytomous Items Using a Common Odds Ratio Estimator. , 2007 .

[11]  I. W. Molenaar,et al.  Rasch models: foundations, recent developments and applications , 1995 .

[12]  Carolin Strobl,et al.  Unbiased split selection for classification trees based on the Gini Index , 2007, Comput. Stat. Data Anal..

[13]  Jürgen Rost,et al.  Rasch Models in Latent Classes: An Integration of Two Approaches to Item Analysis , 1990 .

[14]  Matthias Sutter,et al.  Psychological Pressure in Competitive Environments: Evidence from a Randomized Natural Experiment: Comment , 2010, SSRN Electronic Journal.

[15]  Engelbert Theurl,et al.  Socioeconomic Environment and Mortality: A two-level Decomposition by Sex and Cause of Death , 2010 .

[16]  K. Hornik,et al.  Model-Based Recursive Partitioning , 2008 .

[17]  Gregory R. Hancock,et al.  Advances in Latent Variable Mixture Models , 2007 .

[18]  Adrian E. Raftery,et al.  mclust Version 4 for R : Normal Mixture Modeling for Model-Based Clustering , Classification , and Density Estimation , 2012 .

[19]  Stefan Lang,et al.  Working Papers in Economics and Statistics Modeling House Prices Using Multilevel Structured Additive Regression Modeling House Prices Using Multilevel Structured Additive Regression , 2022 .

[20]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[21]  Anne-Laure Boulesteix,et al.  Maximally Selected Chi‐Square Statistics and Binary Splits of Nominal Variables , 2006, Biometrical journal. Biometrische Zeitschrift.

[22]  Matthias Sutter,et al.  Gender, Competition and the Efficiency of Policy Interventions , 2010, SSRN Electronic Journal.

[23]  Johannes Gehrke,et al.  Bias Correction in Classification Tree Construction , 2001, ICML.

[24]  Jesús Crespo-Cuaresma,et al.  Business cycle convergence in EMU : A first look at the second moment * , 2010 .

[25]  L. Hubert,et al.  Comparing partitions , 1985 .

[26]  Achim Zeileis,et al.  Tests of Measurement Invariance Without Subgroups: A Generalization of Classical Methods , 2013, Psychometrika.

[27]  Josef Baumgartner,et al.  Milking The Prices: The Role of Asymmetries in the Price Transmission Mechanism for Milk Products in Austria , 2010 .

[28]  Dimitris Rizopoulos,et al.  ltm: An R Package for Latent Variable Modeling and Item Response Analysis , 2006 .

[29]  E. B. Andersen,et al.  A goodness of fit test for the rasch model , 1973 .

[30]  Stuart Parkes,et al.  Allgemeinbildung in Deutschland: Erkenntnisse aus dem SPIEGEL-Studentenpisa-Test , 2012 .

[31]  G. Masters.  A rasch model for partial credit scoring , 1982 .

[32]  Matthias Sutter,et al.  University of Innsbruck Working Papers in Economics and Statistics Household Decision Making in Rural China : Using Experiments to Estimate the Influences of Spouses , 2010 .

[33]  Crespo Cuaresma,et al.  Octavio Fernandez-Amador Business cycle convergence in EMU : A second look at the second moment , 2010 .

[34]  K. Hornik,et al.  Unbiased Recursive Partitioning: A Conditional Inference Framework , 2006 .

[35]  Matthias Sutter,et al.  Equality, Equity and Incentives: An Experiment , 2013, SSRN Electronic Journal.

[36]  Edward E. Rigdon,et al.  Advances in Latent Variable Mixture Models , 2010 .

[37]  Carolin Strobl,et al.  Wissen Frauen weniger oder nur das Falsche? Ein statistisches Modell für unterschiedliche Aufgaben-Schwierigkeiten in Teilstichproben , 2010 .

[38]  M. Hanke,et al.  Football championships and jersey sponsors’ stock prices: an empirical investigation , 2013 .

[39]  Matthias Sutter,et al.  Strategic Sophistication of Adolescents: Evidence from Experimental Normal-Form Games , 2010, SSRN Electronic Journal.

[40]  Achim Zeileis,et al.  psychotree - Recursive partitioning based on psychometric models: Version 0.12-1 , 2011 .

[41]  D. Andrews.  Tests for Parameter Instability and Structural Change with Unknown Change Point , 1993 .

[42]  Engelbert Theurl,et al.  Working Papers in Economics and Statistics Stronger Sex but Earlier Death: a Multi-level Socioeconomic Analysis of Gender Differences in Mortality in Austria Stronger Sex but Earlier Death: a Multi-level Socioeconomic Analysis of Gender Differences in Mortality in Austria , 2022 .

[43]  Matthias Sutter,et al.  Teams Make You Smarter: Learning and Knowledge Transfer in Auctions and Markets by Teams and Individuals , 2010, SSRN Electronic Journal.

[44]  Robert J. Mislevy,et al.  Modeling item responses when different subjects employ different solution strategies , 1990 .

[45]  Achim Zeileis,et al.  Testing for Measurement Invariance with Respect to an Ordinal Variable , 2014, Psychometrika.

[46]  K. Hornik,et al.  Generalized M‐fluctuation tests for parameter instability , 2007 .

[47]  Torsten Hothorn,et al.  On the Exact Distribution of Maximally Selected Rank Statistics , 2002, Comput. Stat. Data Anal..

[48]  D. Andrews.  Tests for Parameter Instability and Structural Change with Unknown Change Point , 1993 .

[49]  Ruth M. Pfeiffer,et al.  Working Papers in Economics and Statistics Comparing Penalized Splines and Fractional Polynomials for Flexible Modelling of the Effects of Continuous Predictor Variables , 2010 .

[50]  M. Sutter,et al.  Strategic Sophistication of Individuals and Teams in Experimental Normal-Form Games , 2010, SSRN Electronic Journal.

[51]  P. Raschky,et al.  Working Papers in Economics and Statistics Uncertainty of Governmental Relief and the Crowding out of Insurance Uncertainty of Governmental Relief and the Crowding out of Insurance , 2022 .

[52]  A. Tamhane,et al.  Multiple Comparison Procedures , 2009 .

[53]  D. Siegmund,et al.  Maximally Selected Chi Square Statistics , 1982 .

[54]  Achim Zeileis,et al.  A new method for detecting differential item functioning in the Rasch model , 2011 .

[55]  Henk Kelderman,et al.  Examining differential item functioning due to item difficulty and alternative attractiveness , 1992 .

[56]  Achim Zeileis,et al.  Generalized Maximally Selected Statistics , 2008, Biometrics.

[57]  Gershon Ben-Shakhar,et al.  Gender Differences in Multiple‐Choice Tests: The Role of Differential Guessing Tendencies , 1991 .

[58]  Francis Tuerlinckx,et al.  A nonlinear mixed model framework for item response theory. , 2003, Psychological methods.

[59]  Y.-S. Shih,et al.  A note on split selection bias in classification trees , 2004, Comput. Stat. Data Anal..

[60]  B. Carleton,et al.  The Dimensionality and Gender Differential Item Functioning of the Mini Asthma Quality of Life Questionnaire (MiniAQLQ) , 2004 .

[61]  Henk Kelderman,et al.  The Mixed Birnbaum Model: Estimation using Collateral Information , 2000 .

[62]  George B. Macready,et al.  The Use of Loglinear Models for Assessing Differential Item Functioning Across Manifest and Latent Examinee Groups , 1990 .

[63]  Matthias Sutter,et al.  Social preferences during childhood and the role of gender and age -- An experiment in Austria and Sweden , 2011 .

[64]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[65]  R. Petersen,et al.  Differential item functioning of the Boston Naming Test in cognitively normal African American and Caucasian older adults , 2009, Journal of the International Neuropsychological Society.

[66]  Randall D. Penfield,et al.  Using a Taxonomy of Differential Step Functioning to Improve the Interpretation of DIF in Polytomous Items: An Illustration , 2008 .

[67]  Eric Turkheimer,et al.  Illustration of MIMIC-Model DIF Testing with the Schedule for Nonadaptive and Adaptive Personality , 2009, Journal of psychopathology and behavioral assessment.

[68]  Wim Van Den Noortgate,et al.  Assessing and Explaining Differential Item Functioning Using Logistic Mixed Models , 2005 .

[69]  G. H. Fischer,et al.  The linear logistic test model as an instrument in educational research , 1973 .

[70]  C. McHorney,et al.  Assessment of Differential Item Functioning for Demographic Comparisons in the MOS SF-36 Health Survey , 2006, Quality of Life Research.

[71]  David Huffman,et al.  University of Innsbruck Working Papers in Economics and Statistics Group Membership , Competition , and Altruistic versus Antisocial Punishment : Evidence from Randomly Assigned Army Groups , 2010 .

[72]  Achim Zeileis,et al.  Accounting for Individual Differences in Bradley-Terry Models by Means of Recursive Partitioning , 2011 .

[73]  Jan-Eric Gustafsson,et al.  Testing and obtaining fit of data to the Rasch model , 1980 .

[74]  W. Stout.  Psychometrics: From practice to theory and back , 2002 .

[75]  G. W. Milligan,et al.  A Study of the Comparability of External Criteria for Hierarchical Cluster Analysis. , 1986, Multivariate behavioral research.

[76]  Patrick Mair,et al.  Extended Rasch Modeling , 2015 .

[77]  A. Tamhane,et al.  Multiple Comparison Procedures , 1989 .

[78]  M. Gächter,et al.  Convergence of the Health Status at the Local Level: Empirical Evidence from Austria , 2010 .

[79]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[80]  Stefan Lang,et al.  Working Papers in Economics and Statistics Applications of Multilevel Structured Additive Regression Models to Insurance Data Applications of Multilevel Structured Additive Regression Models to Insurance Data , 2022 .

[81]  P. Mair,et al.  Extended Rasch Modeling: The eRm Package for the Application of IRT Models in R , 2007 .

[82]  Matthias Sutter,et al.  Working Papers in Economics and Statistics Gender differences in competition emerge early in life , 2010 .

[83]  Matthias Sutter,et al.  Working Papers in Economics and Statistics Group polarization in the team dictator game reconsidered , 2007 .

[84]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[85]  Matthias Sutter,et al.  Social Preferences in Childhood and Adolescence: A Large-Scale Experiment , 2010, SSRN Electronic Journal.

[86]  Adrian E. Raftery,et al.  MCLUST Version 3: An R Package for Normal Mixture Modeling and Model-Based Clustering , 2006 .