Believability of relative risks and odds ratios in abstracts: cross sectional study

Abstract Objective To compare the distribution of P values in abstracts of randomised controlled trials with that in observational studies, and to check P values between 0.04 and 0.06. Design Cross sectional study of all 260 abstracts in PubMed of articles published in 2003 that contained “relative risk” or “odds ratio” and reported results from a randomised trial, and random samples of 130 abstracts from cohort studies and 130 from case-control studies. P values were noted or calculated if unreported. Main outcome measures Prevalence of significant P values in abstracts and distribution of P values between 0.04 and 0.06. Results The first result in the abstract was statistically significant in 70% of the trials, 84% of cohort studies, and 84% of case-control studies. Although many of these results were derived from subgroup or secondary analyses, or biased selection of results, they were presented without reservations in 98% of the trials. P values were more extreme in observational studies (P < 0.001) and in cohort studies than in case-control studies (P = 0.04). The distribution of P values around P = 0.05 was extremely skewed. Only five trials had 0.05 ≤ P < 0.06, whereas 29 trials had 0.04 ≤ P < 0.05. I could check the calculations for 27 of these trials. One of four non-significant results was significant. Four of the 23 significant results were wrong, five were doubtful, and four could be discussed. Nine cohort studies and eight case-control studies reported P values between 0.04 and 0.06, but in all 17 cases P < 0.05. Because the analyses had been adjusted for confounders, these results could not be checked. Conclusions Significant results in abstracts are common but should generally be disbelieved.

[1]  P. Holland Statistics and Causal Inference , 1985 .

[2]  S. Assmann,et al.  Subgroup analysis and other (mis)uses of baseline data in clinical trials , 2000, The Lancet.

[3]  D. Altman,et al.  Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research , 2004, Canadian Medical Association Journal.

[4]  P. Gøtzsche Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. , 1989, Controlled clinical trials.

[5]  A H Briggs,et al.  Constructing confidence intervals for cost-effectiveness ratios: an evaluation of parametric and non-parametric techniques using Monte Carlo simulation. , 1999, Statistics in medicine.

[6]  D. Altman,et al.  Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors , 2005, BMJ : British Medical Journal.

[7]  A H Briggs,et al.  Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. , 1997, Health economics.

[8]  P. Gøtzsche,et al.  Sample size of randomized double-blind trials 1976-1991. , 1996, Danish medical bulletin.

[9]  S. Ebrahim,et al.  Data dredging, bias, or confounding , 2002, BMJ : British Medical Journal.

[10]  S. Pocock,et al.  Statistical problems in the reporting of clinical trials. A survey of three medical journals. , 1987, The New England journal of medicine.

[11]  D G Altman,et al.  Statistics Notes: The use of transformation when comparing two means , 1996, BMJ.

[12]  G. Taubes Epidemiology faces its limits. , 1995, Science.

[13]  V. McCormack,et al.  Issues in the reporting of epidemiological studies: a survey of recent practice , 2004, BMJ : British Medical Journal.

[14]  C. Hine,et al.  Do Community Hospitals Reduce the Use of District General Hospital Inpatient beds? , 1996, Journal of the Royal Society of Medicine.

[15]  J M Bland,et al.  Is the clinical trial evidence about new drugs statistically adequate? , 1985, British journal of clinical pharmacology.

[16]  P. Gøtzsche,et al.  Blinding during data analysis and writing of manuscripts. , 1996, Controlled clinical trials.

[17]  C. Seamark,et al.  Community hospitals for the new millennium. , 2001, The British journal of general practice : the journal of the Royal College of General Practitioners.

[18]  S Greenland,et al.  Randomization, Statistics, and Causal Inference , 1990, Epidemiology.

[19]  Jonathan A C Sterne,et al.  Sifting the evidence—what's wrong with significance tests? , 2001, BMJ : British Medical Journal.

[20]  C. Gluud,et al.  Citation bias of hepato-biliary randomized clinical trials. , 2002, Journal of clinical epidemiology.

[21]  E. von Elm,et al.  Full publication of results initially presented in abstracts. , 2007, The Cochrane database of systematic reviews.

[22]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[23]  A. Forster,et al.  Effects of locality based community hospital care on independence in older people needing rehabilitation: randomised controlled trial , 2005, BMJ : British Medical Journal.