IMPLEMENTING SHAFFER'S MULTIPLE COMPARISON PROCEDURE FOR LARGE NUMBERS OF GROUPS

Shaffer (1986) presented a multiple comparison procedure for paired comparisons that had superior power to many available alternatives. Unfortunately, use of the procedure has been severely limited by the complexity of its implementation. This paper presents a method of allowing Shaffer's procedure to be used in a much larger class of problems. Theoretical results concerning an aspect of Shaffer's procedure are derived. The use of these results in implementing the procedure is the described. Next, two heuristics are described which greatly enhance the efficiency of the method. Finally, an efficient algorithmic method of implementing Shaffer's procedure is outlined. The present work allows Shaffer's procedure for multiple comparisons to be applied to large numbers of groups. Unlike earlier work, no assumptions about the testing situation need be made. These methods are illustrated by application to two data sets; comparisons of the order of 11 clustering methods, and comparison of the means of 44 states from the 1994 National Assessment of Educational Progress Trial State Assessment of Reading at Grade 4.

[1]  Charles W. Dunnett,et al.  A Step-Up Multiple Test Procedure , 1992 .

[2]  L. Toothaker.  Multiple Comparisons for Researchers , 1991 .

[3]  John Mazzeo.  Technical Report of the NAEP 1994 Trial State Assessment Program in Reading. , 1995 .

[4]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[5]  J. Shaffer.  Modified Sequentially Rejective Multiple Test Procedures , 1986 .

[6]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[7]  J. L. Rasmussen.  Algorithm for Shaffer'S Multiple Comparison Tests , 1993 .

[8]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[9]  Norman Cliff,et al.  Complete orders from incomplete data: Interactive ordering and tailored testing. , 1975 .

[10]  A. Tamhane,et al.  Multiple Comparison Procedures , 1989 .

[11]  G. Hommel.  A stagewise rejective multiple test procedure based on a modified Bonferroni test , 1988 .

[12]  B. Holland,et al.  Improved Bonferroni-type multiple testing procedures. , 1988 .

[13]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[14]  J. Donoghue.  A Preliminary Comparison of the Effectiveness of Cluster Analysis Weighting Procedures for Within-Group Covariance Structure. , 1995 .

[15]  William M. Rand,et al.  Objective Criteria for the Evaluation of Clustering Methods , 1971 .

[16]  B. Holland,et al.  An Improved Sequentially Rejective Bonferroni Test Procedure , 1987 .

[17]  J. Tukey.  Comparing individual means in the analysis of variance. , 1949, Biometrics.

[18]  Peter H. Westfall,et al.  Multiple Testing of General Contrasts Using Logical Constraints and Correlations , 1997 .

[19]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .