Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing

Consider the problem of testing k hypotheses simultaneously. In this article we discuss finite- and large-sample theory of stepdown methods that provide control of the familywise error rate (FWE). To improve on the Bonferroni method or on Holm's stepdown method, Westfall and Young made effective use of resampling to construct stepdown methods that implicitly estimate the dependence structure of the test statistics. However, their methods depend on an assumption known as “subset pivotality.” Our goal here is to construct general stepdown methods that do not require such an assumption. To accomplish this, we take a close look at what makes stepdown procedures work; a key component is a monotonicity requirement of critical values. By imposing monotonicity on estimated critical values (which is not an assumption on the model but rather is an assumption on the method), we show how to construct stepdown tests that can be applied in a stagewise fashion so that at most k tests need to be computed. Moreover, at ea...

[1]  W. Hoeffding.  The Large-Sample Power of Tests Based on Permutations of Observations , 1952 .

[2]  Murray Aitkin,et al.  Some tests for correlation matrices , 1969 .

[3]  Correction: Some Tests for Correlation Matrices , 1971 .

[4]  K. Gabriel,et al.  On closed testing procedures with special reference to ordered analysis of variance , 1976 .

[5]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[6]  K. Ruben Gabriel,et al.  Multiple comparisons by rerandomization tests , 1983 .

[7]  G. Hommel.  Multiple test procedures for arbitrary dependence structures , 1986 .

[8]  R. Beran.  Simulated Power Functions , 1986 .

[9]  R. Beran.  Prepivoting Test Statistics: A Bootstrap View of Asymptotic Refinements , 1988 .

[10]  Joseph P. Romano.  A Bootstrap Revival of Some Nonparametric Distance Tests , 1988 .

[11]  Rudolf Beran,et al.  Balanced Simultaneous Confidence Sets , 1988 .

[12]  Joseph P. Romano.  Bootstrap and randomization tests of some nonparametric hypotheses , 1989 .

[13]  D. Pollard,et al.  Cube Root Asymptotics , 1990 .

[14]  Joseph P. Romano.  On the behaviour of randomization tests without the group invariance assumption , 1990 .

[15]  Susan R. Wilson,et al.  Two guidelines for bootstrap hypothesis testing , 1991 .

[16]  S. S. Young,et al.  Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment , 1993 .

[17]  J. Booth,et al.  Resampling-Based Multiple Testing. , 1994 .

[18]  J. Troendle,et al.  A Stepwise Resampling Method of Multiple Hypothesis Testing , 1995 .

[19]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[20]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[21]  P. Westfall,et al.  Multiple Tests with Discrete Distributions , 1997 .

[22]  E. Mammen.  The Bootstrap and Edgeworth Expansion , 1997 .

[23]  H. Finner,et al.  Asymptotic comparison of step-down and step-up multiple test procedures based on exchangeable test statistics , 1998 .

[24]  H. Finner,et al.  Stepwise multiple test procedures and control of directional errors , 1999 .

[25]  Joseph P. Romano,et al.  On Subsampling Estimators with Unknown Rate of Convergence , 1999 .

[26]  Xiao-Hua Zhou,et al.  Pairwise comparisons of the means of skewed data , 2000 .

[27]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[28]  Michael Wolf,et al.  Subsampling inference in cube root asymptotics with an application to Manski's maximum score estimator , 2001 .

[29]  Jason C. Hsu,et al.  Multiple Comparisons of Biodiversity , 2001 .

[30]  Dmitri V Zaykin,et al.  Multiple tests for genetic effects in association studies. , 2002, Methods in molecular biology.

[31]  S. Dudoit,et al.  Multiple Hypothesis Testing in Microarray Experiments , 2003 .

[32]  Alan David Hutson,et al.  Resampling Methods for Dependent Data , 2004, Technometrics.

[33]  E. L. Lehmann,et al.  On optimality of stepdown and stepup multiple test procedures , 2005 .