False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters

Often in applied research, confidence intervals (CIs) are constructed or reported only for parameters selected after viewing the data. We show that such selected intervals fail to provide the assumed coverage probability. By generalizing the false discovery rate (FDR) approach from multiple testing to selected multiple CIs, we suggest the false coverage-statement rate (FCR) as a measure of interval coverage following selection. A general procedure is then introduced, offering FCR control at level q under any selection rule. The procedure constructs a marginal CI for each selected parameter, but instead of the confidence level 1 − q being used marginally, q is divided by the number of parameters considered and multiplied by the number selected. If we further use the FDR controlling testing procedure of Benjamini and Hochberg for selecting the parameters, the newly suggested procedure offers CIs that are dual to the testing procedure and are shown to be optimal in the independent case. Under the positive regression dependency condition of Benjamini and Yekutieli, the FCR is controlled for one-sided tests and CIs, as well as for a modification for two-sided testing. Results for general dependency are also given. Finally, using the equivalence of the CIs to testing, we prove that the procedure of Benjamini and Hochberg offers directional FDR control as conjectured.

[1]  J. Pratt.  Length of Confidence Intervals , 1961 .

[2]  R. A. Waller,et al.  A Bayes Rule for the Symmetric Multiple Comparisons Problem , 1969 .

[3]  S. S. Young,et al.  Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment , 1993 .

[4]  B. Mangin,et al.  Constructing confidence intervals for QTL location. , 1994, Genetics.

[5]  L. Brown,et al.  Optimal Confidence Sets, Bioequivalence, and the Limaçon of Pascal , 1995 .

[6]  G A Colditz,et al.  Intake of carotenoids and retinol in relation to risk of prostate cancer. , 1995, Journal of the National Cancer Institute.

[7]  E. Lander,et al.  Genetic dissection of complex traits: guidelines for interpreting and reporting linkage results , 1995, Nature Genetics.

[8]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[9]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[10]  W. Johnson,et al.  A Bayesian perspective on the Bonferroni adjustment , 1997 .

[11]  J. Weller,et al.  A new approach to the problem of multiple comparisons in the genetic dissection of complex traits. , 1998, Genetics.

[12]  John W. Tukey,et al.  Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement , 1999 .

[13]  Juliet Popper Shaffer,et al.  A semi-Bayesian study of Duncan's Bayesian multiple comparison procedure , 1999 .

[14]  Y. Benjamini,et al.  On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics , 2000 .

[15]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[16]  John D. Storey,et al.  Empirical Bayes Analysis of a Microarray Experiment , 2001 .

[17]  Charles Kooperberg,et al.  Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results From the Women's Health Initiative randomized controlled trial. , 2002, JAMA.

[18]  Graham A Colditz,et al.  Failure of estrogen plus progestin therapy for prevention. , 2002, JAMA.

[19]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[20]  J. Shaffer.  Multiplicity, directional (type III) errors, and the null hypothesis. , 2002, Psychological methods.

[21]  L. Wasserman,et al.  Operating characteristics and extensions of the false discovery rate procedure , 2002 .

[22]  S. Dudoit,et al.  Multiple Hypothesis Testing in Microarray Experiments , 2003 .

[23]  Dorothy T. Thayer,et al.  A loss function related to the FDR for random effects multiple comparisons , 2004 .

[24]  John D. Storey,et al.  Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach , 2004 .

[25]  Y. Benjamini,et al.  Quantitative Trait Loci Analysis Using the False Discovery Rate , 2005, Genetics.