Implementing false discovery rate control: increasing your power

Popular procedures to control the chance of making type I errors when multiple statistical tests are performed come at a high cost: a reduction in power. As the number of tests increases, power for an individual test may become unacceptably low. This is a consequence of minimizing the chance of making even a single type I error, which is the aim of, for instance, the Bonferroni and sequential Bonferroni procedures. An alternative approach, control of the false discovery rate (FDR), has recently been advocated for ecological studies. This approach aims at controlling the proportion of significant results that are in fact type I errors. Keeping the proportion of type I errors low among all significant results is a sensible, powerful, and easy-to-interpret way of addressing the multiple testing issue. To encourage practical use of the approach, in this note we illustrate how the proposed procedure works, we compare it to more traditional methods that control the familywise error rate, and we discuss some recent useful developments in FDR control.

[1]  Luis V. García,et al.  Escaping the Bonferroni iron claw in ecological studies , 2004 .

[2]  Y. Hochberg A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[3]  Y. Benjamini,et al.  On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics , 2000 .

[4]  M. Moran Arguments for rejecting the sequential Bonferroni in ecological studies , 2003 .

[5]  E. Spjøtvoll,et al.  Plots of P-values to evaluate many tests simultaneously , 1982 .

[6]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[7]  L. Wasserman,et al.  Operating characteristics and extensions of the false discovery rate procedure , 2002 .

[8]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[9]  W. Rice ANALYZING TABLES OF STATISTICAL TESTS , 1989, Evolution; international journal of organic evolution.

[10]  John D. Storey,et al.  Statistical significance for genomewide studies , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[11]  John D. Storey A direct approach to false discovery rates , 2002 .

[12]  Y. Benjamini,et al.  More powerful procedures for multiple significance testing. , 1990, Statistics in medicine.

[13]  Robert Tibshirani,et al.  The 'miss rate' for the analysis of gene expression data. , 2005, Biostatistics.

[14]  Luis V. García,et al.  Controlling the false discovery rate in ecological research. , 2003 .

[15]  M. A. Black,et al.  A note on the adaptive control of false discovery rates , 2004 .

[16]  B W Brown,et al.  Methods of correcting for multiple testing: operating characteristics. , 1997, Statistics in medicine.

[17]  S. Holm A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[18]  J. Cheverud,et al.  A simple correction for multiple comparisons in interval mapping genome scans , 2001, Heredity.