Using false discovery rates for multiple comparisons in ecology and evolution

Summary 1. Ecologists and evolutionary biologists often need to simultaneously evaluate the significance of multiple related hypotheses. Multiple comparisons need to be corrected to avoid inappropriately increasing the number of null hypotheses that are wrongly rejected. The traditional method of correction involves Bonferroni-type multiple comparison procedures which are highly conservative, tending to increase the number of wrong rejections of true hypotheses as the number of hypotheses being simultaneously tested increases. 2. Newer procedures which are based on False Discovery Rates and which do not suffer the same loss of power as traditional methods are described. Algorithms and spreadsheet-based software routines for three procedures which are especially useful in ecology and evolution are provided. 3. The strengths and potential pitfalls of FDR-based analysis and of presenting results as FDR-adjusted P-values are discussed with reference to traditional methods such as the sequential Bonferroni correction. 4. FDR-based multiple comparison procedures should be more widely adopted because they are often more appropriate than traditional methods for identifying truly significant results.

[1]  Y. Benjamini,et al.  Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics , 1999 .

[2]  D. B. Duncan.  MULTIPLE RANGE AND MULTIPLE F TESTS , 1955 .

[3]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[4]  H. Scheffé.  A METHOD FOR JUDGING ALL CONTRASTS IN THE ANALYSIS OF VARIANCE , 1953 .

[5]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[6]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[7]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[8]  John D. Storey,et al.  Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach , 2004 .

[9]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[10]  Siu Hung Cheung,et al.  Familywise robustness criteria for multiple‐comparison procedures , 2002 .

[11]  Rupert G. Miller.  Simultaneous Statistical Inference , 1966 .

[12]  J. Troendle,et al.  Stepwise normal theory multiple test procedures controlling the false discovery rate , 2000 .

[13]  R. Fisher,et al.  STATISTICAL METHODS AND SCIENTIFIC INDUCTION , 1955 .

[14]  E. Spjøtvoll,et al.  Plots of P-values to evaluate many tests simultaneously , 1982 .

[15]  M. Keuls,et al.  The use of the „studentized range” in connection with an analysis of variance , 1952, Euphytica.

[16]  B. Sorić.  Statistical “Discoveries” and Effect-Size Estimation , 1989 .

[17]  Y. Benjamini,et al.  On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics , 2000 .

[18]  John D. Storey,et al.  Statistical significance for genomewide studies , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[19]  John D. Storey.  The positive false discovery rate: a Bayesian interpretation and the q-value , 2003 .

[20]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[21]  O. J. Dunn.  On multiple tests and confidence intervals , 1974 .

[22]  E. Lehmann.  Testing Statistical Hypotheses , 1960 .

[23]  C. Dunnett.  A Multiple Comparison Procedure for Comparing Several Treatments with a Control , 1955 .

[24]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[25]  Z. Šidák.  Rectangular Confidence Regions for the Means of Multivariate Normal Distributions , 1967 .

[26]  Y. Benjamini,et al.  Adaptive linear step-up procedures that control the false discovery rate , 2006 .

[27]  M. A. Black,et al.  A note on the adaptive control of false discovery rates , 2004 .