Judicious Use of Multiple Hypothesis Tests

Abstract:  When analyzing a table of statistical results, one must first decide whether adjustment of significance levels is appropriate. If the main goal is hypothesis generation or initial screening for potential conservation problems, then it may be appropriate to use the standard comparisonwise significance level to avoid Type II errors (not detecting real differences or trends). If the main goal is rigorous testing of a hypothesis, however, then an adjustment for multiple tests is needed. To control the familywise Type I error rate (the probability of rejecting at least one true null hypothesis), sequential modifications of the standard Bonferroni method, such as Holm's method, will provide more statistical power than the standard Bonferroni method. Additional power may be achieved through procedures that control the false discovery rate (FDR) (the expected proportion of false positives among tests found to be significant). Holm's sequential Bonferroni method and two FDR‐controlling procedures were applied to the results of multiple‐regression analyses of the relationship between habitat variables and the abundance of 25 species of forest birds in Japan, and the FDR‐controlling procedures provided considerably greater statistical power.

[1]  日本鳥学会.  日本鳥類目録 = Check-list of Japanese birds , 1974 .

[2]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[3]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[4]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[5]  G. Hommel.  A stagewise rejective multiple test procedure based on a modified Bonferroni test , 1988 .

[6]  W. Rice.  ANALYZING TABLES OF STATISTICAL TESTS , 1989, Evolution; international journal of organic evolution.

[7]  D. Saville.  Multiple Comparison Procedures: The Practical Solution , 1990 .

[8]  J. Tukey.  The Philosophy of Multiple Comparisons , 1991 .

[9]  S. P. Wright,et al.  Adjusted P-values for simultaneous inference , 1992 .

[10]  J. Shaffer.  Multiple Hypothesis Testing , 1995 .

[11]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[12]  Geoffrey M. Clarke,et al.  Introduction to the Design and Analysis of Experiments , 1996 .

[13]  Kaye E. Basford,et al.  Graphical profiles as an aid to understanding plant breeding experiments , 1997 .

[14]  J. Weller,et al.  A new approach to the problem of multiple comparisons in the genetic dissection of complex traits. , 1998, Genetics.

[15]  J. Crabbe,et al.  Genetics of mouse behavior: interactions with laboratory environment. , 1999, Science.

[16]  Y. Benjamini,et al.  A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence , 1999 .

[17]  John W. Tukey,et al.  Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement , 1999 .

[18]  Richard O. Lynch,et al.  Introduction to Design and Analysis of Experiments , 1999, Technometrics.

[19]  Y. Benjamini,et al.  On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics , 2000 .

[20]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[21]  Y. Benjamini,et al.  Controlling the false discovery rate in behavior genetics research , 2001, Behavioural Brain Research.

[22]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[23]  John D. Storey.  The positive false discovery rate: a Bayesian interpretation and the q-value , 2003 .

[24]  R. Askins,et al.  Effects of Habitat Fragmentation on Birds in Deciduous Forests in Japan , 2003 .

[25]  John D. Storey,et al.  Statistical significance for genomewide studies , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[26]  John D. Storey,et al.  Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach , 2004 .