Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing

Closed testing procedures are classically used for familywise error rate control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion in all subsets of the hypotheses, allowing for inference robust to post hoc selection of subsets. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut and use it to investigate the power of this approach when the number of hypotheses goes to infinity. We show that if a minimal level of signal is present, the average power to detect false hypotheses at any desired false discovery proportion does not vanish. Additionally, we show that the confidence bounds for false discovery proportion are consistent estimators for the true false discovery proportion for every nonvanishing subset. We also show close connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.

[1]  William Fithian,et al.  AdaPT: an interactive procedure for multiple testing with side information , 2016, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[2]  G. Hommel.  Multiple test procedures for arbitrary dependence structures , 1986 .

[3]  Dan Nettleton,et al.  Estimating the number of true null hypotheses from a histogram of p values , 2006 .

[4]  Sanat K. Sarkar,et al.  On the Simes inequality and its generalization , 2008, 0805.2322.

[5]  Iterated limits in *(ⁿ) , 1973 .

[6]  L. Wasserman,et al.  Exceedance Control of the False Discovery Proportion , 2006 .

[7]  Joseph P. Romano,et al.  Generalizations of the familywise error rate , 2005, math/0507420.

[8]  Helmut Finner,et al.  On the Simes test under dependence , 2017 .

[9]  Einar Andreas Rødland,et al.  Simes' procedure is ‘valid on average’ , 2006 .

[10]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[11]  Joseph P. Romano,et al.  Consonance and the Closure Method in Multiple Testing , 2009 .

[12]  J. Goeman,et al.  A shortcut for Hommel's procedure in linearithmic time , 2017, 1710.08273.

[13]  Kevin S S Henning,et al.  Closed Testing in Pharmaceutical Research: Historical and Recent Developments , 2015, Statistics in biopharmaceutical research.

[14]  Zhiyi Chi.  On the performance of FDR control: Constraints and a partial solution , 2007, 0710.3287.

[15]  Jelle J. Goeman,et al.  Multiple hypothesis testing in genomics , 2014, Statistics in medicine.

[16]  Étienne Roquain,et al.  On empirical distribution function of high-dimensional Gaussian vector components with an application to multiple testing , 2012, 1210.2489.

[17]  L. Wasserman,et al.  Operating characteristics and extensions of the false discovery rate procedure , 2002 .

[18]  Aaditya Ramdas,et al.  Towards "simultaneous selective inference": post-hoc bounds on the false discovery proportion , 2018, 1803.06790.

[19]  Weijie J. Su.  The FDR-Linking Theorem. , 2018, 1812.08965.

[20]  G. Blanchard,et al.  Post hoc inference via joint family-wise error rate control , 2017, 1703.02307.

[21]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[22]  G. Hommel.  A stagewise rejective multiple test procedure based on a modified Bonferroni test , 1988 .

[23]  Nicolai Meinshausen,et al.  False Discovery Control for Multiple Tests of Association Under General Dependence , 2006 .

[24]  L. Wasserman,et al.  A stochastic process approach to false discovery control , 2004, math/0406519.

[25]  A. Schwartzman,et al.  The Empirical Distribution of a Large Number of Correlated Normal Variables , 2015, Journal of the American Statistical Association.

[26]  K. Gabriel,et al.  On closed testing procedures with special reference to ordered analysis of variance , 1976 .

[27]  Bradley Efron,et al.  Large-scale inference , 2010 .

[28]  Jelle J. Goeman,et al.  All-Resolutions Inference for brain imaging , 2017, NeuroImage.

[29]  D. Donoho,et al.  Higher criticism for detecting sparse heterogeneous mixtures , 2004, math/0410072.

[30]  Michael Wolf,et al.  Optimal Testing of Multiple Hypotheses with Common Effect Direction , 2008 .

[31]  A. Tamhane,et al.  A class of improved hybrid Hochberg-Hommel type step-up multiple test procedures , 2014 .

[32]  Daniel Yekutieli.  False discovery rate control for non-positively regression dependent test statistics , 2008 .

[33]  T. Dickhaus,et al.  On the Simes inequality in elliptical models , 2014 .

[34]  B. Lindqvist,et al.  Estimating the proportion of true null hypotheses, with application to DNA microarray data , 2005 .

[35]  A. Buja,et al.  Valid post-selection inference , 2013, 1306.1059.

[36]  J. Goeman,et al.  Multiple Testing for Exploratory Research , 2011, 1208.2841.

[37]  K. Athreya,et al.  General Glivenko–Cantelli theorems , 2016 .

[38]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[39]  W. Brannath,et al.  Shortcuts for Locally Consonant Closed Test Procedures , 2010 .