Using a linear mixed-effect model framework to estimate multivariate generalizability theory parameters in R

Multivariate generalizability theory (mG-theory) is an important framework in many behavioral and educational studies, as it describes useful psychometric properties of multidimensional assessments. Nevertheless, the use of mG-theory estimation is limited due to the lack of available software for carrying out the necessary calculations: users rely heavily on independent software programs such as mGENOVA and the BUGS/JAGS suite of programs. Considering the prevalence of R software, this paper presents a solution using the glmmTMB package to accomplish the estimation task. Users adopting the proposed method may find it more convenient for conducting both applied investigation and simulation studies without the need to switch between different software programs.

[1]  Zhehan Jiang,et al.  The Use of Multivariate Generalizability Theory to Evaluate the Quality of Subscores , 2018, Applied psychological measurement.

[2]  G. Joe,et al.  Some developments in multivariate generalizability , 1976 .

[3]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[4]  Edward H. Haertel,et al.  4 Reliability Coefficients and Generalizability Theory , 2006 .

[5]  Daniel Stegmueller,et al.  How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches , 2013 .

[6]  L. Cronbach,et al.  Generalizability of scores influenced by multiple sources of variance , 1965, Psychometrika.

[7]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[8]  L. A. Goodman.  The Analysis of Cross-Classified Data Having Ordered and/or Unordered Categories: Association Models, Correlation Models, and Asymmetry Models for Contingency Tables With or Without Missing Entries , 1985 .

[9]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[10]  Douglas M. Bates,et al.  Estimating the Multilevel Rasch Model: With the lme4 Package , 2007 .

[11]  Charles E. Lance,et al.  A Test of the Context Dependency of Three Causal Models of Halo Rater Error , 1994 .

[12]  J. A. Woodward,et al.  Maximizing the coefficient of generalizability in multi-facet decision studies , 1973 .

[13]  George A. Marcoulides,et al.  Selecting Weighting Schemes in Multivariate Generalizability Studies , 1994 .

[14]  George A. Marcoulides,et al.  Estimating Variance Components in Generalizability Theory: The Covariance Structure Analysis Approach. Teacher's Corner. , 1996 .

[15]  Alan Huebner,et al.  Generalizability Theory in R , 2019 .

[16]  Casper W. Berg,et al.  glmmTMB Balances Speed and Flexibility Among Packages for Zero-inflated Generalized Linear Mixed Modeling , 2017, R J..

[17]  Allan D. Shocker,et al.  Estimating the weights for multiple attributes in a composite criterion using pairwise judgments , 1973 .

[18]  Albert Nußbaum,et al.  Multivariate Generalizability Theory in Educational Measurement: An Empirical Study , 1984 .

[19]  Govind S. Mudholkar,et al.  Generalized Multivariate Estimator for the Mean of Finite Populations , 1967 .

[20]  Nathan T. Carter,et al.  Updating Generalizability Theory in Management Research , 2015 .

[21]  A Multivariate Generalizability Theory Approach to Standard Setting , 2015, Applied psychological measurement.

[22]  Zhehan Jiang,et al.  Improving generalizability coefficient estimate accuracy: A way to incorporate auxiliary information , 2018 .

[23]  M. Raymond,et al.  Indices of Subscore Utility for Individuals and Subgroups Based on Multivariate Generalizability Theory , 2020, Educational and psychological measurement.

[24]  Giorgio F. Gilestro,et al.  Rethomics: An R framework to analyse high-throughput behavioural data , 2018, bioRxiv.

[25]  Julian C. Stanley,et al.  Differential Weighting: A Review of Methods and Empirical Studies1 , 1970 .

[26]  George A. Marcoulides,et al.  Designing Measurement Studies Under Budget Constraints: Controlling Error of Measurement and Power , 1995 .

[27]  Zhehan Jiang,et al.  A Bayesian approach to estimating variance components within a multivariate generalizability theory framework , 2017, Behavior Research Methods.

[28]  Ali Mili,et al.  An empirical study of programming language trends , 2005, IEEE Software.

[29]  Jonathan Baron,et al.  Behavioral Research Data Analysis with R , 2011 .

[30]  Zhehan Jiang.  Using the Linear Mixed-Effect Model Framework to Estimate Generalizability Variance Components in R , 2018, Methodology.

[31]  Jared P. Lander.  R for Everyone: Advanced Analytics and Graphics , 2013 .

[32]  L. Cronbach,et al.  THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY† , 1963 .

[33]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[34]  B. Clauser,et al.  A Multivariate Generalizability Analysis of Data from a Performance Assessment of Physicians' Clinical Skills , 2006 .

[35]  Bozhidar M. Bashkov,et al.  Determining Item Screening Criteria Using Cost-Benefit Analysis. , 2019 .

[36]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[37]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[38]  George A. Marcoulides,et al.  Estimation of Generalizability Coefficients Via a Structural Equation Modeling Approach to Scale Reliability Evaluation , 2006 .

[39]  David Magis,et al.  Computerized adaptive testing with R: Recent updates of the package catR , 2017 .