Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies

Many studies of development interventions are individually unable to provide convincing conclusions because of low statistical significance, small size, limited geographical purview and so forth. Systematic reviews and meta-analysis are forms of research synthesis that combine studies of adequate methodological quality to produce more convincing conclusions. In the social sciences, study designs, types of analysis and methodological quality vary tremendously. Combining these studies for meta-analysis entails more demanding risk of bias assessments to ensure that only studies with largely appropriate methodological characteristics are included, and sensitivity analysis should be performed. In this article, we discuss assessing risk of bias and meta-analysis using such diverse studies.

[1]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[2]  P. Sleight,et al.  Publication bias , 1991, The Lancet.

[3]  S. Keef,et al.  The meta-analysis of partial effect sizes. , 2004, The British journal of mathematical and statistical psychology.

[4]  J. Vandenbroucke.  Is there a hierarchy of methods in clinical research? , 1989, Klinische Wochenschrift.

[5]  A. Nezu,et al.  Evidence-based outcome research: a practical guide to conducting randomized controlled trials for psychosocial interventions , 2007 .

[6]  G. Smith,et al.  Meta-analysis Spurious precision? Meta-analysis of observational studies , 1998, BMJ.

[7]  Gene V. Glass,et al.  Meta-Analysis of Research on Class Size and Achievement , 1979 .

[8]  B. Wampold,et al.  Meta-analysis in the social sciences: , 2000 .

[9]  K. Dickersin,et al.  Publication bias and clinical trials. , 1987, Controlled clinical trials.

[10]  R. Rosenthal.  Meta-analytic procedures for social research , 1984 .

[11]  Steven Glazerman,et al.  Nonexperimental Versus Experimental Estimates of Earnings Impacts , 2003 .

[12]  M. L. Smith,et al.  Meta-analysis of psychotherapy outcome studies. , 1977, The American psychologist.

[13]  Betsy Jane Becker,et al.  Failsafe N or File‐Drawer Number , 2006 .

[14]  S Shapiro,et al.  Meta-analysis/Shmeta-analysis. , 1994, American journal of epidemiology.

[15]  Ingram Olkin,et al.  Stochastically dependent effect sizes. , 1994 .

[16]  G. Cumming.  Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis , 2011 .

[17]  R. Hanka.  The Handbook of Research Synthesis , 1994 .

[18]  Wolfgang Viechtbauer,et al.  Publication bias in meta-analysis: Prevention, assessment and adjustments , 2007, Psychometrika.

[19]  J. Higgins.  Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration , 2011 .

[20]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[21]  M. Egger,et al.  Meta-analyses of observational data should be done with due care , 1999, BMJ.

[22]  David B. Pillemer,et al.  Summing Up: The Science of Reviewing Research , 1984 .

[23]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[24]  Howard S. Bloom,et al.  The Core Analytics of Randomized Experiments for Social Research. MDRC Working Papers on Research Methodology. , 2006 .

[25]  Richard D Riley,et al.  Interpretation of random effects meta-analyses , 2011, BMJ : British Medical Journal.

[26]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[27]  Maxine Greene,et al.  President's Column: Acting Towards the Future , 1981 .

[28]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[29]  D. Jones,et al.  Meta-analysis of observational epidemiological studies: a review. , 1992, Journal of the Royal Society of Medicine.

[30]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[31]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[32]  Martin Ravallion,et al.  Chapter 59 Evaluating Anti-Poverty Programs , 2007 .

[33]  J. Colliver,et al.  Meta‐analysis of quasi‐experimental research: are systematic narrative reviews indicated? , 2008, Medical education.

[34]  S. Raudenbush,et al.  A multivariate mixed linear model for meta-analysis. , 1996 .

[35]  Erik Weber,et al.  Counterfactuals and causal inference: methods and principles for social research , 2008 .

[36]  Hans J. Eysenck,et al.  Meta-Analysis: an Abuse of Research Integration , 1984 .

[37]  M. Lipsey,et al.  The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. , 1993, American Psychologist.

[38]  R. McGrath,et al.  When effect sizes disagree: the case of r and d. , 2006, Psychological methods.

[39]  H. White.  Some Reflections on Current Debates in Impact Evaluation , 2012 .

[40]  Henrik Hansen,et al.  A Comparison of Model-Based and Design-Based Impact Evaluations of Interventions in Developing Countries , 2013 .

[41]  J. Concato,et al.  Randomized, controlled trials, observational studies, and the hierarchy of research designs. , 2000, The New England journal of medicine.

[42]  A R Feinstein,et al.  Meta-analysis: statistical alchemy for the 21st century. , 1995, Journal of clinical epidemiology.

[43]  Harris Cooper,et al.  Statistically Combining Independent Studies: A Meta-Analysis of Sex Differences in Conformity Research , 1979 .

[44]  H. Hansen,et al.  Systematic Reviews: Questions, Methods and Usage , 2013 .

[45]  M. Turshen.  Development as Freedom , 2001 .

[46]  Alex J Sutton,et al.  Performance of the trim and fill method in the presence of publication bias and between‐study heterogeneity , 2007, Statistics in medicine.

[47]  R. Nickerson,et al.  Null hypothesis significance testing: a review of an old and continuing controversy. , 2000, Psychological methods.

[48]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[49]  Donald B. Rubin,et al.  Meta-Analytic Procedures for Combining Studies With Multiple Effect Sizes , 1986 .

[50]  W. Shadish,et al.  Assignment methods in experimentation: When do nonrandomized experiments approximate answers from randomized experiments? , 1996 .

[51]  Julia Kastner,et al.  Introduction to Robust Estimation and Hypothesis Testing , 2005 .

[52]  A. Hartz,et al.  A comparison of observational studies and randomized, controlled trials. , 2000, The New England journal of medicine.

[53]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[54]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[55]  Larry V. Hedges,et al.  The Effects of Class Size: An Examination of Rival Hypotheses , 1983 .

[56]  Jake Bowers,et al.  Covariate balance in simple stratified and clustered comparative studies , 2008, 0808.3857.

[57]  L. Hedges,et al.  A Brief History of Research Synthesis , 2002, Evaluation & the health professions.

[58]  Angus Deaton.  Instruments, Randomization, and Learning about Development , 2010 .

[59]  H. Eysenck.  An exercise in mega-silliness. , 1978 .

[60]  L. Hooper,et al.  Systematic Review. What is the evidence of the impact of microfinance on the well-being of poor people? , 2011 .

[61]  Betsy Jane Becker,et al.  The Synthesis of Regression Slopes in Meta-Analysis. , 2007, 0801.4442.

[62]  D. Sharpe,et al.  Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away. , 1997, Clinical psychology review.

[63]  David R. Jones,et al.  Systematic reviews of trials and other studies. , 1998, Health technology assessment.

[64]  F. Song,et al.  Evaluating non-randomised intervention studies. , 2003, Health technology assessment.

[66]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[67]  Robert E. Slavin,et al.  Best-Evidence Synthesis: An Alternative to Meta-Analytic and Traditional Reviews , 1986 .

[68]  D. Mccloskey,et al.  The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives , 2008 .

[69]  Adriana Petryna,et al.  When Experiments Travel: Clinical Trials and the Global Search for Human Subjects , 2009 .

[70]  Stephen W. Raudenbush,et al.  Analyzing effect sizes: Random-effects models. , 2009 .

[71]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[72]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[73]  M. Kendall.  Statistical Methods for Research Workers , 1937, Nature.

[74]  R. Orwin.  A fail-safe N for effect size in meta-analysis. , 1983 .

[75]  Gary King,et al.  An Introduction to the Dataverse Network as an Infrastructure for Data Sharing , 2007 .

[76]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[77]  M. Ravallion.  Evaluating Anti-Poverty Programs , 2005 .

[78]  I. Olkin,et al.  Meta-analysis of observational studies in epidemiology - A proposal for reporting , 2000 .

[79]  Harris Cooper,et al.  A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD). , 2008, Psychological methods.

[80]  P. Cummings,et al.  The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century , 2001 .

[81]  Michael Kremer,et al.  Chapter 61 Using Randomization in Development Economics Research: A Toolkit ★ , 2007 .

[82]  R. Grissom,et al.  Effect Sizes for Research : Univariate and Multivariate Applications, Second Edition , 2005 .

[83]  Richard A. Berk,et al.  Statistical Assumptions as Empirical Commitments , 2001 .

[84]  Paul D. Ellis,et al.  The essential guide to effect sizes : statistical power, meta-analysis, and the interpretation of research results , 2010 .

[85]  A Pingsmann,et al.  Sample size and statistical power. , 2000, The Journal of bone and joint surgery. American volume.