On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics

A new approach to problems of multiple significance testing was presented in Benjamini and Hochberg (1995), which calls for controlling the expected ratio of the number of erroneous rejections to the number of rejections–the False Discovery Rate (FDR). The procedure given there was shown to control the FDR for independent test statistics. When some of the hypotheses are in fact false, that procedure is too conservative. We present here an adaptive procedure, where the number of true null hypotheses is estimated first as in Hochberg and Benjamini (1990), and this estimate is used in the procedure of Benjamini and Hochberg (1995). The result is still a simple stepwise procedure, to which we also give a graphical companion. The new procedure is used in several examples drawn from educational and behavioral studies, addressing problems in multi-center studies, subset analysis and meta-analysis. The examples vary in the number of hypotheses tested, and the implication of the new procedure on the conclusions. In a large simulation study of independent test statistics the adaptive procedure is shown to control the FDR and have substantially better power than the previously suggested FDR controlling method, which by itself is more powerful than the traditional family wise error-rate controlling methods. In cases where most of the tested hypotheses are far from being true there is hardly any penalty due to the simultaneous testing of many hypotheses.

[1]  C. Daniel.  Locating Outliers in Factorial Experiments , 1960 .

[2]  David R. Cox,et al.  A Remark on Multiple Comparison Methods , 1965 .

[3]  P. Seeger.  A Note on a Method for the Analysis of Significances en masse , 1968 .

[4]  J. Tukey.  Some thoughts on clinical trials, especially problems of multiplicity. , 1977, Science.

[5]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[6]  E. Spjøtvoll,et al.  Plots of P-values to evaluate many tests simultaneously , 1982 .

[7]  N. Victor.  Exploratory Data Analysis and Clinical Research , 1982, Methods of Information in Medicine.

[8]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[9]  K. K. Lan,et al.  Some implications of an alternative definition of the multiple comparison problem , 1988 .

[10]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[11]  G. Hommel.  A stagewise rejective multiple test procedure based on a modified Bonferroni test , 1988 .

[12]  B. Sorić.  Statistical “Discoveries” and Effect-Size Estimation , 1989 .

[13]  A. Tamhane,et al.  Multiple Comparison Procedures , 1989 .

[14]  Y. Benjamini,et al.  More powerful procedures for multiple significance testing. , 1990, Statistics in medicine.

[15]  J. Tukey.  The Philosophy of Multiple Comparisons , 1991 .

[16]  S. Lyness.  Predictors of differences between Type A and B individuals in heart rate and blood pressure reactivity. , 1993, Psychological bulletin.

[17]  Gerhard Hommel,et al.  Multiple Hypotheses Testing , 1993 .

[18]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[19]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[20]  A. Sadeh,et al.  The Reaction of the Youth in Israel to the Assassination of Prime Minister Yitzhak Rabin , 1998 .

[21]  P. Stark,et al.  Confidence Intervals with More Power to Determine the Sign: Two Ends Constrain the Means , 1998 .

[22]  Y. Benjamini,et al.  Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics , 1999 .

[23]  John W. Tukey,et al.  Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement , 1999 .

[24]  P. Sen.  Some remarks on Simes-type multiple tests of significance , 1999 .

[25]  P. Sen.  Multiple comparisons in interim analysis , 1999 .

[26]  J. Troendle,et al.  Stepwise normal theory multiple test procedures controlling the false discovery rate , 2000 .