The citation bias: Fad and fashion in the judgment and decision literature.

is not dependent upon the following two conditions: (a) all zs above some c (e.g., 1.96) are reported in the literature, and (b) all reports in the literature contain zs above some c (e.g., 1.96). Condition b appears to be reasonably accurate (e.g., 37 of the 42 studies we reported have zs above 1.96). Condition a is probably not very accurate because studies can be rejected for reasons other than small zs. The alternative to the null hypothesis is that (because there is some effect of therapy) the distribution of zs—whatever it is—is centered to the right of 0 and hence the zs will be larger than predicted by the null hypothesis. To test this null hypothesis, we constructed 200 random samples of one z-value greater than 1.96 from each of our 37 studies reporting at least one value that large. Thus, there was no within-study dependence between zs (and no reason to expect between-study dependence). The average of the average zs was 2.77, not 2.34 as predicted by the null hypothesis. Because the variance of the normal truncated above 1.96 is .14, the test z comparing 2.77 to 2.34 is 7.56 (.43 divided by (. 14/37)). p is virtually 0. Similar results are found with cut points of 1.65, 2.33, and 2.58. Unfortunately, the results counterindicate going "backwards" from the hypothesized tail to infer the location of the hypothesized mean. The reason is that all the variances of the zs actually computed are four to six times larger than those based on normal curve theory. All we can do is reject—soundly—the null hypothesis, without introducing the "small enough" ambiguity of the Rosenthal method. The discrepancy between theoretical and observed variances mitigates against any normal curve "correction" of effect size. Sampling independent zs above 1.96 thus led to a mean very significantly larger than 2.34, the expected value if we were sampling zs above 1.96 from a unit normal distribution. Our conclusion is that we are not randomly sampling from a truncated normal. Specifically, the zs are larger. The significant effects of psychotherapy cannot be accounted for by selective reporting unless there is an additional bias that the larger the z beyond the standard significance level the more likely it is to be reported. If such a bias existed, we would expect the results above 2.58 to be nonexistent, or at least weaker than those above 1.96. But they are not (test z = 7.42, p virtually 0). Our basic assumption is quite broad. In fact, when c approaches — oo, it is the standard assumption underlying normal distribution significance tests. All we have done is to apply the same logic to a (truncated) part of that distribution.