A Family of Experiments to Investigate the Influence of Context on the Effect of Inspection Techniques

Abstract For a growing population of researchers in software engineering, empirical studies have become a key approach of re-search. Empirical studies may be used, for example, to evaluate technologies and help to direct further research by revealing what problems and difficulties people have in practice. Without empirical studies, we have to rely only on intuition or educated opinion. Individual empirical studies often yield interesting results for their particular context, but typically this context is not de-scribed in sufficient detail to decide whether another context is similar enough to apply the conclusions of the study also there. We argue instead that families of experiments with a common framework for collecting context data are necessary in order to abstract conclusions at a useful level of detail. This paper describes a method to plan, conduct, and analyze coordinated, or concerted, families of experiments. The goal of the method is to maximize the quality and benefit of the individual empirical studies as part of the family and to mini-mize the effort for researchers by reusing experiment know-how. This is achieved by providing, for all studies of the ex-periment family, a common framework for context measurement, study preparation, material, and analysis. We apply the method to describe the planning steps for an experiment family on the influence of context on the effec-tiveness of defect reduction techniques. We focus on a particular technology, reading techniques for inspections, to in-stantiate this work. The first step of this experiment family is a broad survey of software companies on the state of the practice of inspection process and inspection techniques. The second step is to benchmark state-of-the-art inspection techniques with the par-ticipating organization’s own documents and inspection techniques. Keywords: Empirical Software Engineering, Experiment Family, Software Inspection, Meta Analysis. 1 I