Multiple Hypothesis Testing

INTRODUCTION ..................................................................................................................... 561 ORGANIZING CO CEPTS ..................................................................................................... 564 Primary Hypotheses, Closure, Hierarchical Sets, and Minimal Hypotheses ...................... 564 Families ................................................................................................................................ 565 Type 1 Error Control ............................................................................................................ 566 Power ................................................................................................................................... 567 P-Values and Adjusted P-Values ......................................................................................... 568 Closed Test Procedures ....................................................................................................... 569 METHODS BA ED ON ORDERED P-VALUES ................................................................... 569 Methods Based on the First-Order Bonferroni Inequality .................................................. 569 Methods Based on the Simes Equality ................................................................................. 570 Modifications for Logically Related Hypotheses ................................................................. 571 Methods Controlling the False Discovery Rate ................................................................... 572 COMPARING NORMALLY DISTRIBUTED M ANS ......................................................... 573 OTHER ISSUES ........................................................................................................................ 575 Tests vs Confidence I tervals ............................................................................................... 575 Directional vs Nondirectional Inference ............................................................................. 576 Robustness ............................................................................................................................ 577 Others ........................................................................ ....................................................... 578 CONCLUSION .......................................................................................................................... 580

[1]  B. L. Welch.  THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUAL , 1938 .

[2]  F. E. Satterthwaite.  An approximate distribution of estimates of variance components. , 1946, Biometrics.

[3]  Frederick Mosteller,et al.  A $k$-Sample Slippage Test for an Extreme Population , 1948 .

[4]  J. Tukey.  Comparing individual means in the analysis of variance. , 1949, Biometrics.

[5]  E. Paulson.  A Multiple Decision Procedure for Certain Problems in the Analysis of Variance , 1949 .

[6]  H. Scheffé.  A METHOD FOR JUDGING ALL CONTRASTS IN THE ANALYSIS OF VARIANCE , 1953 .

[7]  R. C. Bose,et al.  Simultaneous Confidence Interval Estimation , 1953 .

[8]  C. Dunnett.  A Multiple Comparison Procedure for Comparing Several Treatments with a Control , 1955 .

[9]  D. B. Duncan.  MULTIPLE RANGE AND MULTIPLE F TESTS , 1955 .

[10]  H. O. Hartley,et al.  Some recent developments in analysis of variance , 1955 .

[11]  Clyde Young Kramer,et al.  Extension of multiple range tests to group means with unequal numbers of replications , 1956 .

[12]  E. L. Lehmann,et al.  A Theory of Some Multiple Decision Problems. II , 1957 .

[13]  D. B. Duncan.  Multiple range tests for correlated heteroscedastic means , 1957 .

[14]  T. A. Ryan,et al.  Multiple comparisons in psychological research. , 1959, Psychological bulletin.

[15]  T. A. Ryan,et al.  Significance tests for multiple comparison of proportions, variances, and other statistics. , 1960, Psychological bulletin.

[16]  D. B. Duncan.  Bayes Rules for a Common Multiple Comparisons Problem and Related Student- $t$ Problems , 1961 .

[17]  D. B. Duncan,et al.  A Bayesian Approach to Multiple Comparisons , 1965 .

[18]  P. Seeger.  A Note on a Method for the Analysis of Significances en masse , 1968 .

[19]  K. Gabriel,et al.  Simultaneous test procedures in multivariate analysis of variance , 1968 .

[20]  K. Gabriel,et al.  SIMULTANEOUS TEST PROCEDURES-SOME THEORY OF MULTIPLE COMPARISONS' , 1969 .

[21]  H. Scheffé.  Multiple Testing Versus Multiple Estimation. Improper Confidence Sets. Estimation of Directions and Ratios , 1970 .

[22]  E. Spjøtvoll.  On the Optimality of Some Multiple Comparison Procedures , 1972 .

[23]  K. Gabriel,et al.  A Study of the Powers of Several Methods of Multiple Comparisons , 1975 .

[24]  K. Gabriel,et al.  On closed testing procedures with special reference to ordered analysis of variance , 1976 .

[25]  J. Shaffer.  Multiple comparisons emphasizing selected contrasts: an extension and generalization of Dunnett's procedure. , 1977, Biometrics.

[26]  Ordering ordered parameters , 1977 .

[27]  R. Welsch.  Stepwise Multiple Comparison Procedures , 1977 .

[28]  Rupert G. Miller,et al.  Developments in Multiple Comparisons 1966–1976 , 1977 .

[29]  Philip H. Ramsey.  Power Differences between Pairwise Multiple Comparisons , 1978 .

[30]  E. Lehmann,et al.  Optimum Significance Levels for Multistage Comparison Procedures , 1979 .

[31]  R. Bohrer.  Multiple Three-Decision Rules for Parametric Signs , 1979 .

[32]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[33]  Juliet Popper Shaffer.  Control of Directional Errors with Stagewise Multiple Test Procedures , 1980 .

[34]  R Bohrer,et al.  An optimal multiple decision rule for signs of parameters. , 1980, Proceedings of the National Academy of Sciences of the United States of America.

[35]  C. Dunnett.  Pairwise Multiple Comparisons in the Unequal Variance Case , 1980 .

[36]  Philip H. Ramsey.  Power of univariate pairwise multiple comparison procedures. , 1981 .

[37]  J. Hsu.  Simultaneous Confidence Intervals for all Distances from the "Best" , 1981 .

[38]  Complexity: An Interpretability Criterion for Multiple Comparisons , 1981 .

[39]  K. Ruben Gabriel,et al.  Closure of the Newman-Keuls Multiple Comparisons Procedure , 1981 .

[40]  J. Richmond.  A General Method for Constructing Simultaneous Confidence Intervals , 1982 .

[41]  Mark L. Berenson,et al.  A comparison of severalk sample tests for ordered alternatives in completely randomized designs , 1982 .

[42]  E. Spjøtvoll,et al.  Plots of P-values to evaluate many tests simultaneously , 1982 .

[43]  Gideon Weiss,et al.  On Graphical Procedures for Multiple Comparisons , 1982 .

[44]  I. Olkin,et al.  Adjusting P Values to Account for Selection over Dichotomies , 1983 .

[45]  Donald B. Rubin,et al.  Ensemble-Adjusted p Values , 1983 .

[46]  D. Cani,et al.  Balancing Type I Risk and Loss of Power in Ordered Bonferroni Procedures. , 1984 .

[47]  J. Hsu.  Constrained Simultaneous Confidence Intervals for Multiple Comparisons with the Best , 1984 .

[48]  A Note on Two-Sided Distribution-Free Treatment versus Control Multiple Comparisons , 1984 .

[49]  G. K. Eagleson,et al.  A NEW PROCEDURE FOR ASSESSING LARGE SETS OF CORRELATIONS , 1985 .

[50]  Peter Bauer,et al.  The Application of Hunter's Inequality in Simultaneous Testing , 1985 .

[51]  Eve Bofinger,et al.  Multiple Comparisons and Type III Errors , 1985 .

[52]  G. Hommel.  Multiple test procedures for arbitrary dependence structures , 1986 .

[53]  Y. Hochberg.  Multiple classification rules for signs of parameters , 1986 .

[54]  G. Hommel,et al.  Multiple testing of pairs of one-sided hypotheses , 1986 .

[55]  G. K. Eagleson,et al.  Assessing large sets of rank correlations , 1986 .

[56]  J. Shaffer.  Modified Sequentially Rejective Multiple Test Procedures , 1986 .

[57]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[58]  B. Holland,et al.  An Improved Sequentially Rejective Bonferroni Test Procedure , 1987 .

[59]  Michele Conforti,et al.  Sequentially rejective pairwise testing procedures , 1987 .

[60]  Jason C. Hsu,et al.  On Confidence Sets in Multiple Comparisons , 1988 .

[61]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[62]  G. Hommel.  A stagewise rejective multiple test procedure based on a modified Bonferroni test , 1988 .

[63]  B. Holland,et al.  Improved Bonferroni-type multiple testing procedures. , 1988 .

[64]  Generalized maximum range tests for pairwise comparisons of several populations , 1989 .

[65]  Book Reviews: Multiple Comparison Procedures Y. Hochberg and A. C. Tamhane New York: Wiley, 1987. xxii + 450 pp , 1989 .

[66]  P. Bauer,et al.  Multiple Test Procedures in Clinical Dose Finding Studies , 1989 .

[67]  B. Sorić.  Statistical “Discoveries” and Effect-Size Estimation , 1989 .

[68]  G. Hommel.  A comparison of two modified Bonferroni procedures , 1989 .

[69]  Charles W. Dunnett,et al.  Two-Stage Procedures for Comparing Treatments with a Control Elimination at the First Stage and Estimation at the Second Stage , 1989 .

[70]  On the modified s-method and directional errors , 1990 .

[71]  D. Rom.  A sequentially rejective test procedure based on a modified Bonferroni inequality , 1990 .

[72]  J. Kunert.  On the power of tests for multiple comparison of three normal means , 1990 .

[73]  A probability inequality for ranges and its application to maximum range test procedures , 1990 .

[74]  D. Hoover,et al.  Subset complement addition upper bounds - an improved inclusion-exclusion method , 1990 .

[75]  J. Tukey.  The Philosophy of Multiple Comparisons , 1991 .

[76]  Juliet Popper Shaffer,et al.  Multiple pairwise comparisons of repeated measures means under violation of multisample sphericity , 1991 .

[77]  J. Shaffer.  Probability of directional errors with disordinal (qualitative) interaction , 1991 .

[78]  H. Keselman,et al.  Maximum familywise Type I error rate: The least significant difference, Newman-Keuls, and other multiple comparison procedures. , 1991 .

[79]  D. Naiman,et al.  INCLUSION-EXCLUSION-BONFERRONI IDENTITIES AND INEQUALITIES FOR DISCRETE TUBE-LIKE PROBLEMS VIA EULER CHARACTERISTICS , 1992 .

[80]  Eric R. Ziegel,et al.  Multiple Comparisons, Selection and Applications in Biometry , 1992 .

[81]  S. P. Wright,et al.  Adjusted P-values for simultaneous inference , 1992 .

[82]  Charles W. Dunnett,et al.  A Step-Up Multiple Test Procedure , 1992 .

[83]  G. Hancock,et al.  Power of recent multiple comparison procedures as applied to a complete set of planned orthogonal contrasts. , 1992 .

[84]  Fred M. Hoppe,et al.  Beyond Inclusion-and-Exclusion: Natural Identities for P(exactly t events) and P(at least t events) and Resulting Inequalities , 1993 .

[85]  J. L. Rasmussen.  Algorithm for Shaffer'S Multiple Comparison Tests , 1993 .

[86]  Mario Peruggia,et al.  Graphical Representations of Tukey's Multiple Comparison Method , 1994 .

[87]  Anthony J. Hayter,et al.  On the Relationship between Stepwise Decision Procedures and Confidence Sets , 1994 .

[88]  R. Nowak.  Problems in clinical trials go far beyond misconduct. , 1994, Science.

[89]  Y. Hochberg,et al.  An extended Simes' test , 1994 .

[90]  A generalized family of multiple test procedures , 1994 .

[91]  Joel R. Levin,et al.  A controlled, powerful multiple-comparison strategy for several situations. , 1994 .

[92]  B. Holland,et al.  A new closed multiple testing procedure for hierarchical families of hypotheses , 1995 .

[93]  H. Keselman,et al.  Improved Repeated Measures Stepwise Multiple Comparison Procedures , 1995 .

[94]  Yosef Hochberg,et al.  Extensions of multiple testing procedures based on Simes' test , 1995 .

[95]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .