Measurement Equivalence, Symmetry, Effect Sizes, and Meta-Analysis

In meta-analysis, different measures are often used in different studies, so the meta-analyst must combine effect sizes (ESs) based on different measures with unique and arbitrary metrics. It has been argued that standardization in ESs places them on a common metric, which makes them comparable. However, recent theoretical development and research has challenged this assertion. In this article, especially relevant for social work meta-analysts, I draw from symmetry and mathematical group theory to create a conceptual and theoretical basis for the measurement conditions needed for the invariance and comparability of ESs based on different measures. This is the first time that mathematical group theory arguments have been used in the social work literature on measurement. Symmetry is defined and illustrated, as is mathematical group theory—a theory of symmetry. I demonstrate that measurement equivalence manifests as symmetries between true-score distributions based on different measures, which leads to invariant true-score ESs and measurement-independent meta-analytic results. I also show that the equal reliabilities of observed scores leads to approximate symmetries between observed-score distributions and invariant ESs and approximately symmetric meta-analytic outcomes. I argue that measurement equivalence is necessary for meta-analyses. Implications of these results for social work research synthesis are considered.

[1]  W. Nugent The Interchangeability of Scores From Different Measures and Meta-Analytic Effect Size Comparability , 2012, Journal of the Society for Social Work and Research.

[2]  James Jaccard,et al.  Arbitrary metrics in psychology. , 2006, The American psychologist.

[3]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[4]  N. Schmitt,et al.  Measurement invariance: Review of practice and implications , 2008 .

[5]  Jia He Bias and Equivalence in Cross-Cultural Research , 2014 .

[6]  R. Grissom,et al.  Effect Sizes for Research : Univariate and Multivariate Applications, Second Edition , 2005 .

[7]  Thomas D. Wickens,et al.  The geometry of multivariate statistics , 1994 .

[8]  M. J. Osler Laws and symmetry , 1993 .

[9]  D. Gross,et al.  The role of symmetry in fundamental physics. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[10]  Allen Rubin,et al.  Practitioner's Guide to Using Research for Evidence-Based Practice , 2007 .

[11]  Julio Michael Stern Symmetry, Invariance and Ontology in Physics and Statistics , 2011, Symmetry.

[12]  P. Gorkin Fearless symmetry: Exposing the hidden patterns of numbers , 2008 .

[13]  Karen François,et al.  Symmetry and Measuring: Ways to Teach the Foundations of Mathematics Inspired by Yupiaq Elders , 2019, Journal of Humanistic Mathematics.

[14]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[15]  W. Nugent The Interchangeability of Scores from Different Measures and Meta-Analytic Effect Size Comparability II: A Simulation Study , 2013, Journal of the Society for Social Work and Research.

[16]  Stijn van Dongen,et al.  Metric distances derived from cosine similarity and Pearson and Spearman correlations , 2012, ArXiv.

[17]  F Dan Richard,et al.  Meta-analysis of raw mean differences. , 2003, Psychological methods.

[18]  J. Ruscio,et al.  A probability-based measure of effect size: robustness to base rates and other factors. , 2008, Psychological methods.

[19]  William R. Nugent Construct Validity Invariance and Discrepancies in Meta-Analytic Effect Sizes Based on Different Measures , 2009 .

[20]  M. Lipsey,et al.  The role of method in treatment effectiveness research: evidence from meta-analysis. , 2001, Psychological methods.

[21]  Shu-Kun Lin Symmetry in Science: An Introduction to the General Theory , 1999, Entropy.

[22]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[23]  A. Panahi Threats to Validity: construct-irrelevant variances contributing to performance under-representation on Graduate Record Exam (GRE) , 2014 .

[24]  S. Messick Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[25]  W. Nugent Variability in the Results of Meta-Analysis as a Function of Comparing Effect Sizes Based on Scores From Noncomparable Measures: A Simulation Study , 2017, Educational and psychological measurement.

[26]  Nathan Carter,et al.  Visual Group Theory , 2009 .

[27]  William R. Nugent,et al.  An Empirical Demonstration of the Existence of Measurement Dependence in the Results of a Meta-Analysis , 2019, Journal of the Society for Social Work and Research.