Significance Level and Power

Significance testing, as described in most textbooks, consists in fixing a standard significance level α such as.01 or.05 and rejecting the hypothesis θ = θ0 if a suitable statistic Y exceeds C where Pθo{Y > C} = α. Such a procedure controls the probability of false rejection (error of the first kind) at the desired level α but leaves the power of the test and hence the probability of an error of the second kind to the mercy of the experi-ment. It seems more natural when deciding on a significance level (and this suggestion is certainly not new) to take into account also what power can be achieved with the given experiment. In Section 3 a specific suggestion will be made as to how to balance α against the power β obtainable against the alterna-tives of interest.