Simultaneous Statistical Inference

We introduce the problem of simultaneous statistical inference, with particular emphasis on testing multiple hypotheses. After a historic overview, general notation for the whole work is set up and different sources of multiplicity are distinguished. We define a variety of classical and modern type I and type II error rates in multiple hypotheses testing, analyze some relationships between them, and consider different ways to cope with structured systems of hypotheses. Relationships between multiple testing and other simultaneous statistical inference problems, in particular the construction of confidence regions for multi-dimensional parameters, as well as selection, ranking and partitioning problems, are elucidated. Finally, a general outline of the remainder of the work is given. Simultaneous statistical inference is concerned with the problem of making several decisions simultaneously based on one and the same dataset. In this work, simultaneous statistical decision problems will mainly be formalized by multiple hypotheses andmultiple tests. Not all simultaneous statistical decision problems are given in this formulation in the first place, but they can often be re-formulated in terms of multiple test problems. General relationships between multiple testing and other kinds of simultaneous statistical decision problems will briefly be discussed in Sect. 1.3. Moreover, we will refer to specific connections at respective occasions. For instance, we will elucidate connections between multiple testing and binary classification in Chap.6 and discuss multiple testing methods in the context of model selection in Chap.7. The origins of multiple hypotheses testing can at least be traced back to Bonferroni (1935, 1936). The “Bonferroni correction”(cf. Example 3.1) is a generic method for evaluating several statistical tests simultaneously and ensuring that the probability for at least one type I error is bounded by a pre-defined significance level α. The latter criterion is nowadays referred to as (strong) control of the familywise error rate (FWER) at level α and will be defined formally in Definition 1.2 below. In well-defined model classes, the Bonferroni method can be improved. In the 1950s, especially analysis of variance (ANOVA) models have been studied with respect to multiple comparisons of group-specific means. For instance, Tukey (1953) T. Dickhaus, Simultaneous Statistical Inference, 1 DOI: 10.1007/978-3-642-45182-9_1, © Springer-Verlag Berlin Heidelberg 2014 2 1 The Problem of Simultaneous Inference developed a multiple test for all pairwise comparisons of means in ANOVA models based on the studentized range distribution. Keuls (1952) applied this technique to a ranking problem of ANOVAmeans in an agricultural context. The works of Dunnett (1955, 1964) treated the problem ofmultiple comparisonswith a control group, while Scheffé (1953) provided a method for testing general linear contrasts simultaneously in the ANOVA context.Concepts from multivariate analysis and probability theory, in particular multivariate dependency concepts, have also been used for multiple testing, cf. for instance the works by Šidák (1967, 1968, 1971, 1973). These concepts allow for establishing probability bounds which in turn can be used for adjusting significance levels for multiplicity. We will provide details in Sect. 4.3. While all the aforementioned historical methods lead to single-step tests (meaning that the same, multiplicity-adjusted critical value is used for all test statistics corresponding to the considered tests), the formal introduction of the closed test principle byMarcus et al. (1976) paved the way for stepwise rejective multiple tests (for a detailed description of these different classes of multiple test procedures, see Chap.3). These stepwise rejective tests are often improvements of the classical single-step tests with respect to power, meaning that they allow (on average) for more rejections of false hypotheses while controlling the same type I error criterion (namely, the FWER at a given level of significance). Stepwise rejective FWER-controlling multiple tests have been developed in the late 1970s, the 1980s and early 1990s; see, for example, Holm (1977, 1979), Hommel (1988) (based on Simes (1986)), Hochberg (1988), and Rom (1990). Around this time, the theory of FWER control had reached a high level of sophistication and was treated in the monographs by Hochberg and Tamhane (1987) and Hsu (1996). It is fair to say that a new era of multiple testing began when Benjamini and Hochberg (1995) introduced a new type I error criterion, namely control of the false discovery rate (FDR), see Definition 1.2. Instead of bounding the probability of one or more type I errors, the FDR criterion bounds the expected proportion of false positives among all significant findings, which typically implies to allow for a few type I errors; see also Seeger (1968) and Sorić (1989) for earlier instances of this idea. During the past 20years, simultaneous statistical inference and, in particular, multiple statistical hypothesis testing has become amajor branch of mathematical and applied statistics, cf. Benjamini (2010) for some bibliometric details. Even for experts it is hardly possible to keep track of the exponentially (over time) growing literature in the field. This growing importance is not least due to the data-analytic challenges posed by large-scale experiments in modern life sciences such as, for instance, genetic association studies (cf. Chap.9), gene expression studies (Chap.10), functional magnetic resonance imaging (Chap.11), and brain-computer interfacing (Chap.12). Hence, the present work is attempting to explain some of the most important theoretical basics of simultaneous statistical inference, togetherwith applications in diverse areas of the life sciences. 1.1 Sources of Multiplicity 3 1.1 Sources of Multiplicity The following definition is fundamental for the remainder of this work. Definition 1.1 (Statistical model). A statistical model is a triple (X ,F ,P). In this,X denotes the sample space (the set of all possible observations),F a σ -field on X (the set of all events that we can assign a probability to) and P a family of probability measures on the measurable space (X ,F ). Often, we will write P in the form (Pθ)θ∈Θ , such that the family is indexed by the parameter θ of the model which can take values in the parameter space Θ , where Θ may have infinite dimension. Unless stated otherwise, an observation will be denoted by x ∈ X , and we think of x as the realization of a random variate X which mathematically formalizes the data-generating mechanism. The target of statistical inference is the parameter θ which we regard as the unknown and unobservable state of nature. Once the statistical model for the data-generating process at hand is defined, two general types of resulting multiplicity can be labeled as “oneor twosample problemswithmultiple endpoints” and “k-sample problemswith localized comparisons”, where k > 2, respectively. In oneor twosample problems with multiple endpoints, the sample space is often of the form X = Rm×n . The same n observational units are measured with respect to m different endpoints, where we assumed for ease of presentation that every measurement results in a real number. The transfer to measurements of other type (for instance, allele pairs at genetic loci) is straightforward. For every of them endpoints, an own scientific question can be of interest. On the contrary, in k-sample problemswith localized comparisons, the sample space is typically of the form X = R ∑k i=1 ni , meaning that k > 2 different groups of observational units (for instance, corresponding to k different doses of a drug) are considered, and that ni observations are made in group i , where 1 ≤ i ≤ k. In this, all ∑k i=1 ni measurements concern one and the same endpoint (for instance, a disease status). The scientific questions in the latter case typically relate to differences between the k groups. Multiplicity arises, if not (only) general homogeneity or heterogeneity between the groups shall be assessed, but if differences, if any, are to be localized in the sense that we want to find out which groups are different. Two classical examples are the “all pairs” problem (all m = k(k − 1)/2 pairwise group comparisons are of interest) and the “multiple comparisons with a control” problem (group k is a reference group and all other m = k − 1 groups are to be compared with group k). We will primarily focus on these two kinds of problems. However, it has to be mentioned that they do not cover the whole spectrum of simultaneous statistical inference problems. For instance, flexible (group-sequential and adaptive) study designs induce a different type of multiplicity problem that we will not consider in the present work. Throughout the remainder, we will try to stick to the notation developed in this section: m is the number of comparisons (the multiplicity of the problem), n or a subscripted n denotes a sample size and k refers to the total number of groups in a 4 1 The Problem of Simultaneous Inference k-sample problem or to the dimensionality of the parameter θ . Often, the two latter quantities are identical. 1.2 Multiple Hypotheses Testing In what follows, we (sometimes implicitly) identify statistical hypotheses with nonempty subsets of the parameter space Θ . The tuple (X ,F , (Pθ)θ∈Θ,H ) denotes a multiple test problem, where H = (Hi : i ∈ I ) for an arbitrary index set I defines a family of null hypotheses. The resulting alternative hypotheses are denoted by Ki = Θ \ Hi , i ∈ I . The intersection hypothesis H0 =⋂i∈I Hi will be referred to as global hypothesis. Throughout the work, we assume that H0 is non-empty.With very few exceptions, we will consider the case of finite families of hypotheses, meaning that |I | = m ∈ N. In such cases, we will often writeHm instead ofH and index the hypotheses such