Extracting replicable associations across multiple studies: algorithms for controlling the false discovery rate

In almost every field in genomics, large-scale biomedical datasets are used to report associations. Extracting associations that recur across multiple studies while controlling the false discovery rate is a fundamental challenge. Here, we consider an extension of Efron’s single-study twogroups model to allow joint analysis of multiple studies. We assume that given a set of p-values obtained from each study, the researcher is interested in associations that recur in at least k > 1 studies. We propose new algorithms that differ in how the study dependencies are modeled. We compared our new methods and others using various simulated scenarios. The top performing algorithm, SCREEN (Scalable Cluster-based REplicability ENhancement), is our new algorithm that is based on three stages: (1) clustering an estimated correlation network of the studies, (2) learning replicability (e.g., of genes) within clusters, and (3) merging the results across the clusters using dynamic programming. We applied SCREEN to two real datasets and demonstrated that it greatly outperforms the results obtained via standard meta-analysis. First, on a collection of 29 case-control large-scale gene expression cancer studies, we detected a large up-regulated module of genes related to proliferation and cell cycle regulation. These genes are both consistently up-regulated across many cancer studies, and are well connected in known gene networks. Second, on a recent pan-cancer study that examined the expression profiles of patients with or without mutations in the HLA complex, we detected an active module of up-regulated genes that are related to immune responses. Thanks to our ability to quantify the false discovery rate, we detected thrice more genes as compared to the original study. Our module contains most of the genes reported in the original study, and many new ones. Interestingly, the newly discovered genes are needed to establish the connectivity of the module. Lay Summary When analyzing results from multiple studies, extracting replicated associations is the first step towards making new discoveries. The standard approach for this task is to use meta-analysis methods, which usually make an underlying null hypothesis that a gene has no effect in all 1 ar X iv :1 60 9. 01 11 8v 2 [ st at .M E ] 7 S ep 2 01 6 studies. On the other hand, in replicability analysis we explicitly require that the gene will manifest a recurring pattern of effects. In this study we develop new algorithms for replicability analysis that are both scalable (i.e., can handle many studies) and allow controlling the false discovery rate. We show that our main algorithm called SCREEN (Scalable Cluster-based REplicability ENhancement) outperforms the other methods in simulated scenarios. Moreover, when applied to real datasets, SCREEN greatly extended the results of the meta-analysis, and can even facilitate detection of new biological results. Introduction Confidence in reported findings is a prerequisite for advancing any scientific field. Such confidence is achieved by showing replication of discoveries by further evidence from new studies [1]. In recent years, a new type of methodology called replicability analysis, sometimes referred to as reproducibility analysis, was suggested as a way to statistically quantify the replication of discoveries across studies while controlling for the false discovery rate (FDR) [2]. This type of analysis is crucial in studies that aim to detect new hypotheses by integrating existing data from multiple high-throughput experiments. The practical importance of replicability analysis is twofold. First, it is a tool for quantifying replication, and therefore the reliability, of reported results. This is of vital importance as in recent years concerns have been raised in several domains regarding low reproducibility, including economics [3], psychology [4], medicine [5], and biological studies that rely on high throughput experiments such as gene expression profiling [6,7], and network biology [8]. Second, collating information from multiple studies can lead to scientific results that may be beyond the reach of a single study. Indeed, replicability analysis was demonstrated as a tool for extracting new results by merging Genome Wide Association Studies (GWAS) [9]. The underlying assumption in standard meta-analysis is that the multiple studies estimate the same effect. Aggregating information across studies produces estimators with smaller measurement error that yield considerably more power to reject the null hypothesis regarding this effect. While meta-analyses are widely applied and have been extensively studied in the statistical literature [10] and in computational biology [11, 12], in recent years the changes in the scale and also the scope of public high-throughput biomedical data has led to new methodological challenges. For example, Zeggini et al. [13] analyzed results of genome-wide association scans for Type 2 Diabetes (T2D) on the same set of almost 2.5 million SNPs from eight study populations. In such situations, the first, and more obvious, challenge is accounting for inflation in the number of false discoveries due to the multiplicity of outcomes. The second challenge is hidden in the null hypothesis that the effect size is 0 in all the studies (as done in meta-analysis). That assumption is oblivious to the consistency of the effects, and thus it overlooks important scientific information. Third, there is a need to distinguish between true effects that are specific to a single study and true effects that represent general discoveries that are replicable. For example, Kraft et al. [14] suggested that for common genetic