Sequential multiple testing with generalized error control: An asymptotic optimality theory

The sequential multiple testing problem is considered under two generalized error metrics. Under the first one, the probability of at least $k$ mistakes, of any kind, is controlled. Under the second, the probabilities of at least $k_1$ false positives and at least $k_2$ false negatives are simultaneously controlled. For each formulation, the optimal expected sample size is characterized, to a first-order asymptotic approximation as the error probabilities go to 0, and a novel multiple testing procedure is proposed and shown to be asymptotically efficient under every signal configuration. These results are established when the data streams for the various hypotheses are independent and each local log-likelihood ratio statistic satisfies a certain Strong Law of Large Numbers. In the special case of i.i.d. observations in each stream, the gains of the proposed sequential procedures over fixed-sample size schemes are quantified.

[1]  John D. Storey.  The optimal discovery procedure: a new approach to simultaneous significance testing , 2007 .

[2]  Georgios Fellouris,et al.  Logarithmically efficient simulation for misclassification probabilities in sequential multiple testing , 2016, 2016 Winter Simulation Conference (WSC).

[3]  Jay Bartroff,et al.  Multiple Hypothesis Tests Controlling Generalized Error Rates for Sequential Data , 2014, 1406.5933.

[4]  Matthew Malloy,et al.  Sequential Testing for Sparse Recovery , 2012, IEEE Transactions on Information Theory.

[5]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[6]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[7]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[8]  Michael Wolf,et al.  Control of generalized error rates in multiple testing , 2007, 0710.2258.

[9]  Edsel A. Peña,et al.  POWER-ENHANCED MULTIPLE DECISION FUNCTIONS CONTROLLING FAMILY-WISE ERROR AND FALSE DISCOVERY RATES. , 2009, Annals of statistics.

[10]  Wenguang Sun,et al.  Large‐scale multiple testing under dependence , 2009 .

[11]  R. H. Farrell.  Limit Theorems for Stopped Random Walks III , 1966 .

[12]  Shyamal K. De,et al.  Step-up and step-down methods for testing multiple hypotheses in sequential experiments , 2012 .

[13]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[14]  T. Lai.  Asymptotic Optimality of Invariant Sequential Probability Ratio Tests , 1981 .

[15]  R. Khan,et al.  Sequential Tests of Statistical Hypotheses. , 1972 .

[16]  Baum–Katz–Nagaev type results for martingales , 2007 .

[17]  Allan Gut,et al.  Limit Theorems for Stopped Random Walks , 1988 .

[18]  Joseph P. Romano,et al.  Generalizations of the familywise error rate , 2005, math/0507420.

[19]  Alexander G. Tartakovsky,et al.  Multichannel Sequential Detection—Part I: Non-i.i.d. Data , 2016, IEEE Transactions on Information Theory.

[20]  H. Robbins,et al.  The Expected Sample Size of Some Tests of Power One , 1974 .

[21]  Alexander G. Tartakovsky,et al.  Asymptotic Optimality of Certain Multihypothesis Sequential Tests: Non‐i.i.d. Case , 1998 .

[22]  Georgios Fellouris,et al.  Asymptotically optimal, sequential, multiple testing procedures with prior information on the number of signals , 2016, 1603.02791.

[23]  Venugopal V. Veeravalli,et al.  Multihypothesis sequential probability ratio tests - Part II: Accurate asymptotic expansions for the expected sample size , 2000, IEEE Trans. Inf. Theory.

[24]  Wenge Guo,et al.  Further results on controlling the false discovery proportion , 2014, 1406.0266.

[25]  I. Pavlov.  Sequential Procedure of Testing Composite Hypotheses with Applications to the Kiefer–Weiss Problem , 1991 .

[26]  M. Basseville,et al.  Sequential Analysis: Hypothesis Testing and Changepoint Detection , 2014 .

[27]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[28]  G. Hommel.  A stagewise rejective multiple test procedure based on a modified Bonferroni test , 1988 .

[29]  E. L. Lehmann,et al.  On optimality of stepdown and stepup multiple test procedures , 2005 .

[30]  Michael Baron,et al.  Sequential Bonferroni Methods for Multiple Hypothesis Testing with Strong Control of Family-Wise Error Rates I and II , 2012 .

[31]  P. Armitage.  Sequential Analysis with More than Two Alternative Hypotheses, and its Relation to Discriminant Function Analysis , 1950 .

[32]  Tze Leung Lai,et al.  Asymptotic approximations for error probabilities of sequential or fixed sample size tests in exponential families , 1999 .

[33]  Joseph P. Romano,et al.  Stepup procedures for control of generalizations of the familywise error rate , 2006, math/0611266.

[34]  X. Rong Li,et al.  Sequential detection of targets in multichannel systems , 2003, IEEE Trans. Inf. Theory.

[35]  Jay Bartroff,et al.  Multistage Tests of Multiple Hypotheses , 2010, 1107.1919.

[36]  Venugopal V. Veeravalli,et al.  Multihypothesis sequential probability ratio tests - Part I: Asymptotic optimality , 1999, IEEE Trans. Inf. Theory.

[37]  G. Lorden.  Open-ended tests for Koopman-Darmois families , 1973 .

[38]  Z. Ying,et al.  Generalized Sequential Probability Ratio Test for Separate Families of Hypotheses , 2014, Sequential analysis.

[39]  P. Varshney,et al.  Multisensor surveillance systems : the fusion perspective , 2003 .

[40]  K. Gabriel,et al.  On closed testing procedures with special reference to ordered analysis of variance , 1976 .

[41]  J. Bartroff,et al.  Sequential Tests of Multiple Hypotheses Controlling Type I and II Familywise Error Rates. , 2013, Journal of statistical planning and inference.

[42]  Michael Baron,et al.  Sequential tests controlling generalized familywise error rates , 2015 .

[43]  H Robbins,et al.  Complete Convergence and the Law of Large Numbers. , 1947, Proceedings of the National Academy of Sciences of the United States of America.