Accuracy of Effect Size Estimates from Published Psychological Research

A Monte-Carlo simulation was used to model the biasing of effect sizes in published studies. The findings from the simulation indicate that, when a predominant bias to publish studies with statistically significant results is coupled with inadequate statistical power, there will be an overestimation of effect sizes. The consequences such an effect size overestimation will then have on meta-analyses and power analyses are highlighted and discussed along with measures which can be taken to reduce the problem.

[1]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[2]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[3]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[4]  David M. Lane,et al.  Estimating effect size: Bias resulting from the significance criterion in editorial decisions , 1978 .

[5]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[6]  Alfred L. Brophy,et al.  Approximation of the inverse normal distribution function , 1985 .

[7]  J. Rossi,et al.  Statistical power of psychological research: what have we gained in 20 years? , 1990, Journal of consulting and clinical psychology.

[8]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[9]  H. Kraemer.  To increase power in randomized clinical trials without increasing sample size. , 1991, Psychopharmacology bulletin.

[10]  Frank L. Schmidt,et al.  What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. , 1992 .

[11]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[12]  G. Loftus.  Psychology Will Be a Much Better Science When We Change the Way We Analyze Data , 1996 .

[13]  Robert P. Abelson,et al.  On the Surprising Longevity of Flogged Horses: Why There Is a Case for the Significance Test , 1997 .

[14]  Patrick E. Shrout,et al.  Should Significance Tests be Banned? Introduction to a Special Section Exploring the Pros and Cons , 1997 .

[15]  J. Hunter.  Needed: A Ban on the Significance Test , 1997 .

[16]  Helena C. Kraemer,et al.  Advantages of excluding underpowered studies in meta-analysis: Inclusionist versus exclusionist viewpoints. , 1998 .

[17]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[18]  Gerd Gigerenzer,et al.  Do Studies of Statistical Power Have an Effect on the Power of Studies? , 2004 .

[19]  M. Birnbaum.  Human research and data collection via the internet. , 2004, Annual review of psychology.

[20]  S. Maxwell.  The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[21]  M. T. Bradley,et al.  Diagnosing Estimate Distortion Due to Significance Testing in Literature on Detection of Deception , 2004, Perceptual and motor skills.

[22]  G. Cumming,et al.  Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think , 2004, Psychological science.

[23]  H. Cooper,et al.  Finding the Missing Science : The Fate of Studies Submitted for Review by a Human Subjects Committee , 2004 .