False-positive results in clinical trials: multiple significance tests and the problem of unreported comparisons.

At least three factors increase the probability that the results in a report of a clinical trial will be false positive: 1) Publication bias—the tendency for investigators to submit and editors to accept selectively reports of trials that appear to give positive results (1-4), some of which will occur by statistical chance. 2) The low probability that new treatments will lead to therapeutic advances (5), implying a low prevalence of truepositive trials (6). Just as a screening procedure (e.g., fecal occult blood test for colorectal cancer) will give a high rate of false-positive results when applied to a population with low prevalence of a disease, so a clinical trial will have a high false-positive rate (and a low false-negative rate) if there is a low prevalence of true differences in outcome. This effect is illustrated by the following example, provided by Dr. M. Parmar (personal communication). Suppose 200 trials are undertaken with a significance level of 5% (type I error of a = 0.05) and with a 0.9 power ( 1 0 [type II error = 0.1]). If the prevalence of trials with a real difference is 10% (20 trials). 18 (20 x 0.9) trials will be reported correctly as true positives; in contrast, there will be nine (180 x 0.05) false positives or one in three (i.e., nine of 27) trials reported as positive. Overall, 173 of the 200 trials will be reported as negative (171 true negatives and two Table 1. Characteristics of 32 reports of randomized clinical trials