Effect size analysis

When we seek insight in collected data we are most often forced to limit our measurements to a portion of all individuals that can be hypothetically considered for observation. Nevertheless, as researchers, we want to draw more general conclusions that are valid beyond the restricted subset we are currently analyzing. Statistical significance testing is a fundamental pattern of data analysis that helps us to infer conclusions from a subset about the entire set of possible individuals. However, the outcome of such tests depends on several factors. Software engineering experiments often address similar research questions but vary with respect to those factors, for example, they operate on different sizes or measurements. Hence, the use of statistical significance alone to interpret findings across studies is insufficient. This paper describes how significance testing can be extended by an analysis of the magnitude, i.e., effect size, of an observation allowing to abstract the results of different studies.