Three alternative procedures to adjust significance levels for multiplicity are the traditional Bonferroni technique, a sequential Bonferroni technique developed by Hochberg (1988), and a sequential approach for controlling the false discovery rate proposed by Benjamini and Hochberg (1995). These procedures are illustrated and compared using examples from the National Assessment of Educational Progress (NAEP). A prominent advantage of the Benjamini and Hochberg (B-H) procedure, as demonstrated in these examples, is the greater invariance of statistical significance for given comparisons over alternative family sizes. Simulation studies show that all three procedures maintain a false discovery rate bounded above, often grossly, by α (or α/2). For both uncorrelated and pairwise families of comparisons, the B-H technique is shown to have greater power than the Hochberg or Bonferroni procedures, and its power remains relatively stable as the number of comparisons becomes large, giving it an increasing advantage when many comparisons are involved. We recommend that results from NAEP State Assessments be reported using the B-H technique rather than the Bonferroni procedure.
[1]
Y. Hochberg.
A sharper Bonferroni procedure for multiple tests of significance
,
1988
.
[2]
John H. Schuenemeyer.
Fundamentals of Exploratory Analysis of Variance
,
1993
.
[3]
Eugene G. Johnson.
Technical Report of the NAEP 1992 Trial State Assessment Program in Mathematics.
,
1993
.
[4]
Ina V. S. Mullis,et al.
NAEP 1992 Mathematics Report Card for the Nation and the States.
,
1993
.
[5]
Y. Benjamini,et al.
Controlling the false discovery rate: a practical and powerful approach to multiple testing
,
1995
.
[6]
Eric R. Ziegel,et al.
Multiple Comparisons, Selection and Applications in Biometry
,
1992
.
[7]
J. Tukey.
The Philosophy of Multiple Comparisons
,
1991
.
[8]
A. Tamhane,et al.
Multiple Comparison Procedures
,
1989
.
[9]
F. Mosteller,et al.
Fundamentals of exploratory analysis of variance
,
1991
.