Sequential Multiple Hypothesis Testing with Type I Error Control

This work studies multiple hypothesis testing in the setting when we obtain data sequentially and may choose when to stop sampling. We summarize the notion of a sequential pvalue (one that can be continually updated and still maintain a type I error guarantee) and provide several examples from the literature. This tool allows us to convert fixedhorizon step-up or step-down multiple hypothesis testing procedures (which includes Benjamini-Hochberg, Holm, and Bonferroni) into a sequential version that allows the statistician to reject a hypothesis as soon as the sequential p-value reaches a threshold while maintaining type I error control. We show that if the original procedure has a type I error guarantee in a certain family (including FDR and FWER), then the sequential conversion inherits an analogous guarantee. The conversion also allows for allocating samples in a data-dependent way, and we provide simulated experiments demonstrating an increased number of rejections when compared to the fixed-horizon setting.

[1]  R. Khan,et al.  Sequential Tests of Statistical Hypotheses. , 1972 .

[2]  H. Robbins,et al.  The Expected Sample Size of Some Tests of Power One , 1974 .

[3]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[4]  T. Lai.  Incorporating scientific, ethical and economic considerations into the design of clinical trials in the pharmaceutical industry: a sequential approach , 1984 .

[5]  H. Robbins,et al.  A Class of Stopping Rules for Testing Parametric Hypotheses , 1985 .

[6]  D. Siegmund.  Sequential Analysis: Tests and Confidence Intervals , 1985 .

[7]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[8]  W. Dupont,et al.  Power and sample size calculations. A review and computer program. , 1990, Controlled clinical trials.

[9]  Venugopal V. Veeravalli,et al.  A sequential procedure for multihypothesis testing , 1994, IEEE Trans. Inf. Theory.

[10]  L. Wasserman,et al.  A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion , 1995 .

[11]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[12]  Christian Homburg,et al.  Applications of structural equation modeling in marketing and consumer research: A review , 1996 .

[13]  Venugopal V. Veeravalli,et al.  Multihypothesis sequential probability ratio tests - Part I: Asymptotic optimality , 1999, IEEE Trans. Inf. Theory.

[14]  Venugopal V. Veeravalli,et al.  Multihypothesis sequential probability ratio tests - Part II: Accurate asymptotic expansions for the expected sample size , 2000, IEEE Trans. Inf. Theory.

[15]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[16]  Joseph P. Romano,et al.  Generalizations of the familywise error rate , 2005, math/0507420.

[17]  Shinichi Nakagawa.  A farewell to Bonferroni: the problems of low statistical power and publication bias , 2004, Behavioral Ecology.

[18]  Peter Bauer,et al.  Two-stage designs for experiments with a large number of hypotheses , 2005, Bioinform..

[19]  S. Sarkar.  False discovery and false nondiscovery rates in single-step multiple testing procedures , 2006, math/0605607.

[20]  S. Sarkar.  STEPUP PROCEDURES CONTROLLING GENERALIZED FWER AND GENERALIZED FDR , 2007, 0803.2934.

[21]  P. Bauer,et al.  Optimized multi‐stage designs controlling the false discovery or the family‐wise error rate , 2008, Statistics in medicine.

[22]  Andrey Novikov,et al.  Optimal Sequential Multiple Hypothesis Tests , 2008, Kybernetika.

[23]  Jay Bartroff,et al.  Multistage Tests of Multiple Hypotheses , 2010, 1107.1919.

[24]  G. Shafer,et al.  Test Martingales, Bayes Factors and p-Values , 2009, 0912.4269.

[25]  A. Tartakovsky,et al.  Almost optimal sequential tests of discrete composite hypotheses , 2012, 1204.5291.

[26]  Shyamal K. De,et al.  Step-up and step-down methods for testing multiple hypotheses in sequential experiments , 2012 .

[27]  J. Bartroff,et al.  Sequential Tests of Multiple Hypotheses Controlling Type I and II Familywise Error Rates. , 2013, Journal of statistical planning and inference.

[28]  L. Pekelis,et al.  Always Valid Inference: Bringing Sequential Analysis to A/B Testing , 2015, 1512.04922.

[29]  Aaditya Ramdas,et al.  Sequential Nonparametric Testing with the Law of the Iterated Logarithm , 2015, UAI.

[30]  J. Bartroff,et al.  Sequential tests of multiple hypotheses controlling false discovery and nondiscovery rates , 2013, Sequential analysis.