Evaluating variability at the software architecture level: an overview

Software architecture are designed for developing software systems needed for a diverse of business goals. Consequently, architecture has to deal with a significant amount of variability in functionality and quality attributes to create different products. Due to this variability, the evaluation in software architectures is much more complex, as different alternatives of systems might be developed leading to an expensive and time consuming task. Several methods and techniques have been proposed to evaluate product line architectures (PLAs) aiming to asses whether or not the architecture will lead to the desired quality attributes. However, there is little consensus on the existing evaluations methods is most suitable for evaluating variability in software architectures, instead of only considering PLAs. Understanding and explicitly evaluating variations in architectures is a cost-effective way of mitigating substantial risk to organizations and their software systems. Therefore, the main contribution of this research work is to present the state of the art about means for evaluating software architectures (including, PLAs, software architectures, reference and enterprise architectures) that contain variability information. We conducted a Systematic Mapping Study (SMS) to provide an overview and insight to practitioners about the most relevant techniques and methods developed for this evaluation. Results indicate that most evaluation techniques assess variability as a quality attribute in PLAs through scenario-based; however, little is known about their real effectiveness as most studies present gaps and lack of evaluation, which difficult the usage of such techniques in an industrial environment.