Handling Complex Meta-analytic Data Structures Using Robust Variance Estimates: a Tutorial in R

PurposeIdentifying and understanding causal risk factors for crime over the life-course is a key area of inquiry in developmental criminology. Prospective longitudinal studies provide valuable information about the relationships between risk factors and later criminal offending. Meta-analyses that synthesize findings from these studies can summarize the predictive strength of different risk factors for crime, and offer unique opportunities for examining the developmental variability of risk factors. Complex data structures are common in such meta-analyses, whereby primary studies provide multiple (dependent) effect sizes.MethodsThis paper describes a recent innovative method for handling complex meta-analytic data structures arising due to dependent effect sizes: robust variance estimation (RVE). We first present a brief overview of the RVE method, describing the underlying models and estimation procedures and their applicability to meta-analyses of research in developmental criminology. We then present a tutorial on implementing these methods in the R statistical environment, using an example meta-analysis on risk factors for adolescent delinquency.ResultsThe tutorial demonstrates how to estimate mean effect sizes and meta-regression models using the RVE method in R, with particular emphasis on exploring developmental variation in risk factors for crime and delinquency. The tutorial also illustrates hypothesis testing for meta-regression coefficients, including tests for overall model fit and incremental hypothesis tests.ConclusionsThe paper concludes by summarizing the benefits of using the RVE method with complex meta-analytic data structures, highlighting how this method can advance research syntheses in the field of developmental criminology.

[1]  Wim Van den Noortgate,et al.  Meta-analysis of multiple outcomes: a multilevel approach , 2014, Behavior Research Methods.

[2]  Lorraine Mazerolle,et al.  Street-level drug law enforcement: A meta-analytical review⋆ , 2007 .

[3]  Shalini Santhakumaran,et al.  Preterm Birth and the Metabolic Syndrome in Adult Life: A Systematic Review and Meta-analysis , 2013, Pediatrics.

[4]  G. Shields,et al.  Does cortisol influence core executive functions? A meta-analysis of acute cortisol administration effects on working memory, inhibition, and set-shifting , 2015, Psychoneuroendocrinology.

[5]  Spyros Konstantopoulos,et al.  Fixed effects and variance components estimation in three‐level meta‐analysis , 2011, Research synthesis methods.

[6]  Dimitris Mavridis,et al.  A practical introduction to multivariate meta-analysis , 2013, Statistical methods in medical research.

[7]  Marvin D. Krohn,et al.  The Development of Delinquency , 2001 .

[8]  H. Kreft,et al.  Environmental heterogeneity as a universal driver of species richness across taxa, biomes and spatial scales. , 2014, Ecology letters.

[9]  Richard D Riley,et al.  An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. , 2008, Biostatistics.

[10]  J. Ioannidis,et al.  Evolution and Translation of Research Findings: From Bench to Where? , 2006, PLoS clinical trials.

[11]  Hannah R Rothstein,et al.  A basic introduction to fixed‐effect and random‐effects models for meta‐analysis , 2010, Research synthesis methods.

[12]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[13]  D. Fuchs,et al.  A Meta-Analysis of Working Memory Deficits in Children With Learning Difficulties , 2016, Journal of learning disabilities.

[14]  D. Weisburd,et al.  The Effects of 'Pulling Levers' Focused Deterrence Strategies on Crime , 2012 .

[15]  Robert D. Abbott,et al.  Assessing Correlates of Onset, Escalation, Deescalation, and Desistance of Delinquent Behavior , 1999 .

[16]  David P. Farrington,et al.  School bullying as a predictor of violence later in life: A systematic review and meta-analysis of prospective longitudinal studies , 2012 .

[17]  David P. Farrington,et al.  Does neighborhood watch reduce crime? A systematic review and meta-analysis , 2007 .

[18]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[19]  David B. Wilson,et al.  A meta-analytic review of court-mandated batterer intervention programs: Can courts affect abusers’ behavior? , 2005 .

[20]  Patrick Onghena,et al.  Multilevel Meta-Analysis: A Comparison with Traditional Meta-Analytical Procedures , 2003 .

[21]  R. Loeber,et al.  Developmental Criminology Updated , 1998, Crime and Justice.

[22]  J. Sánchez-Meca,et al.  Averaging Dependent Effect Sizes in Meta-Analysis: a Cautionary Note about Procedures , 1999, The Spanish Journal of Psychology.

[23]  S. Hansell,et al.  Curriculum Tracking and Delinquency. , 1982 .

[24]  Ingram Olkin,et al.  Stochastically dependent effect sizes. , 1994 .

[25]  Alex R. Piquero,et al.  Self-Control Interventions for Children Under Age 10 for Improving Self-Control and Delinquency and Problem Behaviors: A Systematic Review , 2010 .

[26]  D. Farrington,et al.  Risk and promotive effects in the explanation of persistent serious delinquency in boys. , 2002, Journal of consulting and clinical psychology.

[27]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[28]  Hadley Wickham,et al.  ggplot2 - Elegant Graphics for Data Analysis (2nd Edition) , 2017 .

[29]  John E. Hunter,et al.  Methods of Meta-Analysis , 1989 .

[30]  David Goldblatt,et al.  The Differential Impact of Coadministered Vaccines, Geographic Region, Vaccine Product and Other Covariates on Pneumococcal Conjugate Vaccine Immunogenicity , 2013, The Pediatric infectious disease journal.

[31]  David B. Wilson,et al.  Drug Courts' Effects on Criminal Offending for Juveniles and Adults , 2012 .

[32]  Betsy Jane Becker,et al.  Multivariate meta-analysis. , 2000 .

[33]  Elizabeth Tipton,et al.  Small sample adjustments for robust variance estimation with meta-regression. , 2015, Psychological methods.

[34]  Emily E Tanner-Smith,et al.  Brief alcohol interventions for adolescents and young adults: a systematic review and meta-analysis. , 2015, Journal of substance abuse treatment.

[35]  Richard D Riley,et al.  Rejoinder to commentaries on ‘Multivariate meta‐analysis: Potential and promise’ , 2011 .

[36]  Dena A. Pastor,et al.  Using Mixed-Effects Models In Reliability Generalization Studies , 2003 .

[37]  James E. Pustejovsky,et al.  Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression , 2015 .

[38]  Dan Jackson,et al.  Multivariate meta-analysis: Potential and promise , 2011, Statistics in medicine.

[39]  Elizabeth Tipton,et al.  REG_SANDWICH: Stata module to compute cluster-robust (sandwich) variance estimators with small-sample corrections for linear regression , 2017 .

[40]  Alex R. Piquero,et al.  Violence in criminal careers: A review of the literature from a developmental life-course perspective , 2012 .

[41]  Wim Van den Noortgate,et al.  Three-level meta-analysis of dependent effect sizes , 2012, Behavior Research Methods.

[42]  Mark W. Lipsey,et al.  Risk Factors and Crime , 2012 .

[43]  David P. Farrington,et al.  DEVELOPMENTAL AND LIFE-COURSE CRIMINOLOGY: KEY THEORETICAL AND EMPIRICAL ISSUES-THE 2002 SUTHERLAND AWARD ADDRESS* , 2003 .

[44]  Jeremy W Coid,et al.  Development of adolescence-limited, late-onset, and persistent offenders from age 8 to age 48. , 2009, Aggressive behavior.

[45]  Elizabeth Tipton,et al.  Mindfulness Based Stress Reduction (MBSR) for Improving Health, Quality of Life, and Social Functioning in Adults , 2012 .

[46]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[47]  Elizabeth Tipton,et al.  Erratum: Robust variance estimation in meta‐regression with dependent effect size estimates , 2010, Research synthesis methods.

[48]  Ariel M. Aloe,et al.  Inaccuracy of regression results in replacing bivariate correlations , 2015, Research synthesis methods.

[49]  Elizabeth Tipton,et al.  Robust variance estimation in meta‐regression with binary dependent effects , 2013, Research synthesis methods.

[50]  Gavin B. Stewart,et al.  Assessing the impact of human trampling on vegetation: a systematic review and meta-analysis of experimental evidence , 2014, PeerJ.

[51]  Elizabeth Tipton,et al.  Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and spss , 2014, Research synthesis methods.

[52]  Rolf Holmqvist,et al.  Can psychotherapists function as their own controls? Meta-analysis of the crossed therapist design in comparative psychotherapy trials. , 2013, The Journal of clinical psychiatry.

[53]  Delbert S. Elliott,et al.  Delinquency and dropout , 1974 .

[54]  David P. Farrington,et al.  Life-Course and Developmental Theories in Criminology , 2010 .

[55]  Mike W-L Cheung,et al.  Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. , 2014, Psychological methods.

[56]  Donald B. Rubin,et al.  Meta-Analytic Procedures for Combining Studies With Multiple Effect Sizes , 1986 .