Can meta-analysis be trusted?

Until around 25 years ago the only way to assimilate and evaluate research evidence was through discursive literature reviews, in which someone with an interest in a given research topic would accumulate and subjectively evaluate the importance of research findings in that area. These reviews, although informative, are highly reliant on the discretion of the author who, with the best will in the world, could be unaware of important findings or could give particular importance to studies that others might believe to be relatively less important (see Wolf, 1986). The failure of literature reviews to provide objective ways to assimilate scientific evidence led scientists to look a statistical solution. The groundbreaking work of Glass (1976) and Rosenthal and Rubin (1978) paved the way for what we now know as meta-analysis: a statistical technique by which findings from independent studies can be assimilated.

[1]  Andy P. Field,et al.  Discovering Statistics Using SPSS for Windows: Advanced Techniques for Beginners , 2000 .

[2]  Graham K. Rand,et al.  Quantitative Applications in the Social Sciences , 1983 .

[3]  Donald B. Rubin,et al.  Interpersonal expectancy effects: the first 345 studies , 1978, Behavioral and Brain Sciences.

[4]  J. Feldman.  Meta-Analysis: Cumulating Research Findings Across Studies , 1984 .

[5]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[6]  Gwenn W. Gröndal,et al.  Meta-analytic procedures for social research , 1993 .

[7]  A. Greenwald.  Consequences of Prejudice Against the Null Hypothesis , 1975 .

[8]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[9]  B. Spilker,et al.  Meta-Analysis for Explanation: A Casebook , 1993 .

[10]  John E. Hunter,et al.  Comparison of three meta-analysis methods revisited : An analysis of Johnson, Mullen, and Salas (1995) , 1999 .

[11]  D. P. Gaver,et al.  Combining Information: Statistical Issues and Opportunities for Research , 1993 .

[12]  F. A. Drummond.  C8 , 1970, 1968, Formelregister, Teil 1.

[13]  John E. Hunter,et al.  Fixed Effects vs. Random Effects Meta‐Analysis Models: Implications for Cumulative Research Knowledge , 2000 .

[14]  F. Schmidt,et al.  Measurement Error in Psychological Research: Lessons From 26 Research Scenarios , 1996 .

[15]  A. Field.  Is the meta-analysis of correlation coefficients accurate when population correlations vary? , 2005, Psychological methods.

[16]  K. Dickersin,et al.  Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. , 1992, JAMA.

[17]  Larry V. Hedges,et al.  Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences , 1984 .

[18]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[19]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[20]  Blair T. Johnson,et al.  Comparison of three major meta-analytic approaches. , 1995 .

[21]  A. Field.  Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. , 2001, Psychological methods.

[22]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[23]  A. Field,et al.  The Problems in Using Fixed-Effects Models of Meta-Analysis on Real-World Data , 2003 .

[24]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[25]  Graham J Hole,et al.  How to Design and Report Experiments , 2002 .

[26]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[27]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .