A mixed-method approach for the empirical evaluation of the issue-based variability modeling

Background: Variability management is the fundamental part of software product line engineering, which deals with customization and reuse of artifacts for developing a family of systems. Rationale approaches structure decision-making by managing the tacit-knowledge behind decisions. This paper reports a quasi-experiment for evaluating a rationale enriched collaborative variability management methodology called issue-based variability modeling. Objective: We studied the interaction of stakeholders with issue-based modeling to evaluate its applicability in requirements engineering teams. Furthermore, we evaluated the reuse of rationale while instantiating and changing variability. Approach: We enriched a quasi-experimental design with a variety of methods found in case study research. A sample of 258 students was employed with data collection and analysis based on a mix of qualitative and quantitative methods. Our study was performed in two phases: the first phase focused on variability identification and instantiation, while the second phase included tasks on variability evolution. Results: We obtained strong empirical evidence on reuse patterns for rationale during instantiation and evolution of variability. The tabular representations used by rationale modeling are learnable and usable in teams of diverse backgrounds.