Enriching Meta-Analytic Models of Summary Data: A Thought Experiment and Case Study

Meta-analysis typically involves the analysis of summary data (e.g., means, standard deviations, and sample sizes) from a set of studies via a statistical model that is a special case of a hierarchical (or multilevel) model. Unfortunately, the common summary-data approach to meta-analysis used in psychological research is often employed in settings where the complexity of the data warrants alternative approaches. In this article, we propose a thought experiment that can lead meta-analysts to move away from the common summary-data approach to meta-analysis and toward richer and more appropriate summary-data approaches when the complexity of the data warrants it. Specifically, we propose that it can be extremely fruitful for meta-analysts to act as if they possess the individual-level data from the studies and consider what model specifications they might fit even when they possess only summary data. This thought experiment is justified because (a) the analysis of the individual-level data from the studies via a hierarchical model is considered the “gold standard” for meta-analysis and (b) for a wide variety of cases common in meta-analysis, the summary-data and individual-level-data approaches are, by a principle known as statistical sufficiency, equivalent when the underlying models are appropriately specified. We illustrate the value of our thought experiment via a case study that evolves across five parts that cover a wide variety of data settings common in meta-analysis.

[1]  John W. Tukey,et al.  Analyzing data: Sanctification or detective work? , 1969 .

[2]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[3]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[4]  Elizabeth Tipton,et al.  Current practices in meta‐regression in psychology, education, and medicine , 2019, Research synthesis methods.

[5]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[6]  Hristos Doucouliagos,et al.  What Meta-Analyses Reveal About the Replicability of Psychological Research , 2018, Psychological bulletin.

[7]  Ingram Olkin,et al.  Stochastically dependent effect sizes. , 1994 .

[8]  Ingram Olkin,et al.  Estimation of a Single Effect Size: Parametric and Nonparametric Methods , 1985 .

[9]  Robin Thompson,et al.  [That BLUP is a Good Thing: The Estimation of Random Effects]: Comment , 1991 .

[10]  F Dan Richard,et al.  Meta-analysis of raw mean differences. , 2003, Psychological methods.

[11]  R. Hanka.  The Handbook of Research Synthesis , 1994 .

[12]  Eric-Jan Wagenmakers,et al.  UvA-DARE ( Digital Academic Repository ) Estimates of Between-Study Heterogeneity for 705 Meta-Analyses Reported in Psychological Bulletin From 1990 – 2013 , 2017 .

[13]  Kerrie Mengersen,et al.  Multivariate meta‐analysis , 2003, Statistics in medicine.

[14]  D. Harville.  Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems , 1977 .

[15]  Ulf Böckenholt,et al.  Single Paper Meta-Analysis: Benefits For Study Summary, Theory-Testing, and Replicability , 2017 .

[16]  Harris Cooper,et al.  The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. , 2009, Psychological methods.

[17]  A. Gelman,et al.  Large-Scale Replication Projects in Contemporary Psychological Research , 2017, The American Statistician.

[18]  S. Raudenbush,et al.  A multivariate mixed linear model for meta-analysis. , 1996 .

[19]  Ulf Böckenholt,et al.  Multilevel Multivariate Meta-analysis with Application to Choice Overload , 2018, Psychometrika.

[20]  John A. Johnson.  Measuring thirty facets of the Five Factor Model with a 120-item public domain inventory: Development of the IPIP-NEO-120 , 2014 .

[21]  T. Baguley.  Standardized or simple effect size: what should be reported? , 2009, British journal of psychology.

[22]  Reijo Sund,et al.  Meta‐Analysis: A Structural Equation Modeling Approach , 2016 .

[23]  G. Robinson.  That BLUP is a Good Thing: The Estimation of Random Effects , 1991 .

[24]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[25]  S Greenland,et al.  The fallacy of employing standardized regression coefficients and correlations as measures of effect. , 1986, American journal of epidemiology.

[26]  Donald B. Rubin,et al.  Meta-Analytic Procedures for Combining Studies With Multiple Effect Sizes , 1986 .

[27]  A B Haidich,et al.  Meta-analysis in medical research. , 2010, Hippokratia.

[28]  F Mosteller,et al.  Meta-analysis of multiple outcomes by regression with random effects. , 1998, Statistics in medicine.

[29]  Steven D. Brown,et al.  Handbook of applied multivariate statistics and mathematical modeling , 2000 .

[30]  Betsy Jane Becker,et al.  Modeling multivariate effect sizes. , 1988 .

[31]  L. Stewart,et al.  To IPD or not to IPD? , 2002, Evaluation & the health professions.

[32]  Mark C Simmonds,et al.  Meta-analysis of individual patient data from randomized trials: a review of methods used in practice , 2005, Clinical trials.