SCORE Study Report 8: Closed Tests for All Pairwise Comparisons of Means

We compare five closed tests for strong control of family-wide type 1 error while making all pairwise comparisons of means in clinical trials with multiple arms such as the SCORE Study. We simulated outcomes of the SCORE Study under its design hypotheses, and used P values from chi-squared tests to compare performance of a pairwise closed test described below to Bonferroni and Hochberg adjusted P values. Pairwise closed testing was more powerful than Hochberg's method by several definitions of multiple-test power. Simulations over a wider parameter space, and considering other closed methods, confirmed this superiority for P values based on normal, logistic, and Poisson distributions. The power benefit of pairwise closed testing begins to disappear with five or more arms and with unbalanced designs. For trials with four or fewer arms and balanced designs, investigators should consider using pairwise closed testing in preference to Shaffer's, Hommel's, and Hochberg's approaches when making all pairwise comparisons of means. If not all P values from the closed family are available, Shaffer's method is a good choice.

[1]  Office on Aging.  Manual of policies and procedures , 2009 .

[2]  S. P. Wright,et al.  Adjusted P-values for simultaneous inference , 1992 .

[3]  G. Hommel.  A comparison of two modified Bonferroni procedures , 1989 .

[4]  J. Booth,et al.  Resampling-Based Multiple Testing. , 1994 .

[5]  N. J. A. Sloane,et al.  The On-Line Encyclopedia of Integer Sequences , 2003, Electron. J. Comb..

[6]  Optimal Testing of Multiple Hypotheses with Common Effect Direction , 2008 .

[7]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[8]  Clyde Young Kramer,et al.  Extension of multiple range tests to group means with unequal numbers of replications , 1956 .

[9]  Peter H. Westfall,et al.  Multiple Testing of General Contrasts Using Logical Constraints and Correlations , 1997 .

[10]  Yosef Hochberg,et al.  Closed procedures are better and often admit a shortcut , 1999 .

[11]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[12]  B. Holland,et al.  An Improved Sequentially Rejective Bonferroni Test Procedure , 1987 .

[13]  G. Hommel.  A stagewise rejective multiple test procedure based on a modified Bonferroni test , 1988 .

[14]  J. Neher.  A problem of multiple comparisons , 2011 .

[15]  Z. Šidák.  Rectangular Confidence Regions for the Means of Multivariate Normal Distributions , 1967 .

[16]  Michael Wolf,et al.  Optimal Testing of Multiple Hypotheses with Common Effect Direction , 2008 .

[17]  John R. Donoghue,et al.  IMPLEMENTING SHAFFER'S MULTIPLE COMPARISON PROCEDURE FOR LARGE NUMBERS OF GROUPS , 1998 .

[18]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[19]  H. Scheffé.  A METHOD FOR JUDGING ALL CONTRASTS IN THE ANALYSIS OF VARIANCE , 1953 .

[20]  J. Shaffer.  Modified Sequentially Rejective Multiple Test Procedures , 1986 .

[21]  Allan Birnbaum,et al.  Combining Independent Tests of Significance , 1954 .

[22]  D. Rom.  A sequentially rejective test procedure based on a modified Bonferroni inequality , 1990 .

[23]  P. Westfall,et al.  Multiple Testing of General Contrasts , 2007 .

[24]  K. Gabriel,et al.  On closed testing procedures with special reference to ordered analysis of variance , 1976 .