False discovery rate control for non-positively regression dependent test statistics

Abstract In this paper we present a modification of the Benjamini and Hochberg false discovery rate controlling procedure for testing non-positive dependent test statistics. The new testing procedure makes use of the same series of linearly increasing critical values. Yet, in the new procedure the set of p -values is divided into subsets of positively dependent p -values, and each subset of p -values is separately sorted and compared to the series of critical values. In the first part of the paper we introduce the new testing methodology, discuss the technical issues needed to apply the new approach, and apply it to data from a genetic experiment. In the second part of the paper we discuss pairwise comparisons. We introduce FDR controlling procedures for testing pairwise comparisons. We apply these procedures to an example extensively studied in the statistical literature, and to test pairwise comparisons in gene expression data. We also use the new testing procedure to prove that the Simes procedure can, in some cases, be used to test all pairwise comparisons. The control over the FDR has proven to be a successful alternative to control over the family wise error rate in the analysis of large data sets; the Benjamini and Hochberg procedure has also made the application of the Simes procedure to test the complete null hypothesis unnecessary. Our main message in this paper is that a more conservative approach may be needed for testing non-positively dependent test statistics: apply the Simes procedure to test the complete null hypothesis; if the complete null hypothesis is rejected apply the new testing approach to determine which of the null hypotheses are false. It will probably yield less discoveries, however it ensures control over the FDR.

[1]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[2]  Yoav Benjamini,et al.  Identifying differentially expressed genes using false discovery rate controlling procedures , 2003, Bioinform..

[3]  Martin Vingron,et al.  Variance stabilization applied to microarray data calibration and to the quantification of differential expression , 2002, ISMB.

[4]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[5]  R. Simes,et al.  An improved Bonferroni procedure for multiple tests of significance , 1986 .

[6]  L. W. Erdman,et al.  Studies to Determine if Antibiosis Occurs among Rhizobia: I. between Rhizobium Meliloti and Rhizobium Trifolii 1 , 1946 .

[7]  S. Sarkar,et al.  The Simes Method for Multiple Hypothesis Testing with Positively Dependent Test Statistics , 1997 .

[8]  R. Spang,et al.  Predicting the clinical status of human breast cancer by using gene expression profiles , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Yosef Hochberg,et al.  Improved Bonferroni procedures for testing overall and pairwise homogeneity hypotheses , 1995 .

[10]  P. Seeger.  A Note on a Method for the Analysis of Significances en masse , 1968 .

[11]  Robert A. Cribbie,et al.  The pairwise multiple comparison multiplicity problem: An alternative approach to familywise and comparison wise Type I error control. , 1999 .

[12]  Y. Benjamini,et al.  False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters , 2005 .

[13]  John W. Tukey,et al.  Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement , 1999 .

[14]  S. Sarkar.  Some probability inequalities for ordered $\rm MTP\sb 2$ random variables: a proof of the Simes conjecture , 1998 .

[15]  E. Lehmann.  Some Concepts of Dependence , 1966 .