Quantifying Subpopulation Differences for a Lack of Invariance Using Complex Examinee Profiles: An Exploratory Multigroup Approach Using Functional Data Analysis

This article presents a novel exploratory multigroup approach that quantifies relative group differences within an item response theory framework using tools from functional data analysis. Specifically, examinee groups are formed using different clustering methodologies based on background and attitudinal variable profiles. Item parameters for the groups are then calibrated separately and linked onto a common scale for comparison. Using non-parametric splines and tools from functional data analysis, groups are plotted in a two-dimensional principal components space and their relative difference is interpreted with respect to the different group profiles; each step in the methodology is illustrated with data from the 1999 TIMSS study.

[1]  James O. Ramsay,et al.  Applied Functional Data Analysis: Methods and Case Studies , 2002 .

[2]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[3]  Adrian E. Raftery,et al.  How Many Clusters? Which Clustering Method? Answers Via Model-Based Cluster Analysis , 1998, Comput. J..

[4]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[5]  J. Vermunt,et al.  Latent class cluster analysis , 2002 .

[6]  G. Engelhard.  Historical Views of Invariance: Evidence from the Measurement Theories of Thorndike, Thurstone, and Rasch , 1992 .

[7]  Allan S. Cohen,et al.  Detection of Differential Item Functioning in Multiple Groups. , 1995 .

[8]  André A. Rupp,et al.  Item Response Modeling With BILOG-MG and MULTILOG for Windows , 2003 .

[9]  M. Reckase.  Unifactor Latent Trait Models Applied to Multifactor Tests: Results and Implications , 1979 .

[10]  Charles E. Heckler,et al.  Applied Multivariate Statistical Analysis , 2005, Technometrics.

[11]  Martha L. Stocking,et al.  Developing a Common Metric in Item Response Theory , 1982 .

[12]  Pavel Berkhin,et al.  A Survey of Clustering Data Mining Techniques , 2006, Grouping Multidimensional Data.

[13]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[14]  Glen R. Budgell,et al.  Analysis of Differential Item Functioning in Translated Assessment Instruments , 1995 .

[15]  Jeffrey K. Smith,et al.  Consequence of Performance, Test, Motivation, and Mentally Taxing Items , 1995 .

[16]  Raymond J. Adams,et al.  Multilevel Item Response Models: An Approach to Errors in Variables Regression , 1997 .

[17]  P. Fayers.  Item Response Theory for Psychologists , 2004, Quality of Life Research.

[18]  I. Kostin,et al.  The Subskills of Reading: Rule‐space Analysis of a Multiple‐choice Test of Second Language Reading Comprehension , 1997 .

[19]  Barbara M Byrne,et al.  Measurement equivalence: a comparison of methods based on confirmatory factor analysis and item response theory. , 2002, The Journal of applied psychology.

[20]  E. Klieme,et al.  Identifying national cultures of mathematics education: Analysis of cognitive demands and differential item functioning in TIMSS , 2001 .

[21]  Richard A. Olshen,et al.  CART: Classification and Regression Trees , 1984 .

[22]  B. Zumbo,et al.  An Empirical Test of Roskam's Conjecture about the Interpretation of an ICC Parameter in Personality Inventories , 1997 .

[23]  Luc T. Le.  Analysis of Differential Item Functioning , 2006 .

[24]  R. Brennan,et al.  Test equating : methods and practices , 1995 .

[25]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.

[26]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[27]  William Meredith,et al.  The role of factorial invariance in modeling growth and change. , 2001 .

[28]  W. Zhu.  Test equating: what, why, how? , 1998, Research quarterly for exercise and sport.

[29]  B. Junker.  Some statistical models and computational methods that may be useful for cognitively-relevant assessment , 1999 .

[30]  Paula Garcia,et al.  Combining Multiple Regression and CART to Understand Difficulty in Second Language Reading and Listening Comprehension Test Items , 2001 .

[31]  Jeffrey K. Smith,et al.  The Consequence of Consequence: Motivation, Anxiety, and Test Performance , 1995 .

[32]  J. Ramsay,et al.  Some Tools for Functional Data Analysis , 1991 .

[33]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[34]  Pieter Reitsma,et al.  Educational and Psychological Measurement , 2003 .

[35]  J. Fox,et al.  Bayesian estimation of a multilevel IRT model using gibbs sampling , 2001 .

[36]  P. J. Ferrando.  Calibration of Invariant Item Parameters in a Continuous Item Response Model Using the Extended Lisrel Measurement Submodel. , 1996, Multivariate behavioral research.

[37]  Cornelis A.W. Glas,et al.  Differential Item Functioning Depending on General Covariates , 2001 .

[38]  R. J. Mokken,et al.  Handbook of modern item response theory , 1997 .

[39]  Daniel P. Fasulo,et al.  An Analysis of Recent Work on Clustering Algorithms , 1999 .

[40]  L. J. Stricker,et al.  Possible Determinants of Differential Item Functioning: Familiarity, Interest, and Emotional Reaction. , 1999 .

[41]  Brian E. Clauser,et al.  Using Statistical Procedures to Identify Differentially Functioning Test Items , 2005 .

[42]  Keith F Widaman,et al.  Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. , 1993, Psychological bulletin.

[43]  M. J. Subkoviak,et al.  The Effect of Item Parameter Drift on Examinee Ability Estimates , 2002 .

[44]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[45]  Ronald K. Hambleton,et al.  Identifying the causes of DIF in translated verbal items , 1999 .

[46]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .