Controlling the familywise error rate with plug‐in estimator for the proportion of true null hypotheses

Summary.  Estimation of the number or proportion of true null hypotheses in multiple‐testing problems has become an interesting area of research. The first important work in this field was performed by Schweder and Spjøtvoll. Among others, they proposed to use plug‐in estimates for the proportion of true null hypotheses in multiple‐test procedures to improve the power. We investigate the problem of controlling the familywise error rate FWER when such estimators are used as plug‐in estimators in single‐step or step‐down multiple‐test procedures. First we investigate the case of independent p‐values under the null hypotheses and show that a suitable choice of plug‐in estimates leads to control of FWER in single‐step procedures. We also investigate the power and study the asymptotic behaviour of the number of false rejections. Although step‐down procedures are more difficult to handle we briefly consider a possible solution to this problem. Anyhow, plug‐in step‐down procedures are not recommended here. For dependent p‐values we derive a condition for asymptotic control of FWER and provide some simulations with respect to FWER and power for various models and hypotheses.

[1]  E. Spjøtvoll,et al.  Plots of P-values to evaluate many tests simultaneously , 1982 .

[2]  Y. Benjamini,et al.  More powerful procedures for multiple significance testing. , 1990, Statistics in medicine.

[3]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[4]  J. Swanepoel.  The limiting behavior of a modified maximal symmetric $2s$-spacing with applications , 1999 .

[5]  Y. Benjamini,et al.  On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics , 2000 .

[6]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[7]  H. Finner,et al.  Multiple hypotheses testing and expected number of type I. errors , 2002 .

[8]  M. A. Black,et al.  A note on the adaptive control of false discovery rates , 2004 .

[9]  L. Wasserman,et al.  A stochastic process approach to false discovery control , 2004, math/0406519.

[10]  Peter Bühlmann,et al.  Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures , 2005 .

[11]  I. Johnstone,et al.  Adapting to unknown sparsity by controlling the false discovery rate , 2005, math/0505374.

[12]  B. Lindqvist,et al.  Estimating the proportion of true null hypotheses, with application to DNA microarray data , 2005 .

[13]  T. Cai,et al.  Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple Comparisons , 2006, math/0611108.

[14]  Y. Benjamini,et al.  Adaptive linear step-up procedures that control the false discovery rate , 2006 .

[15]  N. Meinshausen,et al.  Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses , 2005, math/0501289.

[16]  T. Dickhaus,et al.  Dependency and false discovery rate: Asymptotics , 2007, 0710.3171.

[17]  Wenguang Sun,et al.  Oracle and Adaptive Compound Decision Rules for False Discovery Rate Control , 2007 .

[18]  Jiashun Jin.  Proportion of non‐zero normal means: universal oracle equivalences and uniformly consistent estimators , 2008 .