On the number of significant findings to be expected by chance

When multiple significance tests are computed, a certain number of “significant” findings will emerge simply because of chance fluctuations. In the present paper, some factors affecting the number of nominally significant results are elaborated and a general method is suggested which permits unbiased inference as to the significance of a set of findings,as a set. The method advocated employs a high speed computer to generate empirically a sampling distribution tailormade to a particular data matrix. The method is illustrated in the case of dichotomous response to inventory items, where it is found that the statistical model still often used as a basis for estimation is overly conservative. Some problems in the application of the method are discussed.