Gathering current knowledge about quality evaluation in software product lines

Recently, a number of methods and techniques for assessing the quality of software product lines have been proposed. However, to the best of our knowledge, there is no study which summarizes all the existing evidence about them. This paper presents a systematic review that investigates what methods and techniques have been employed (in the last 10 years) to evaluate the quality of software product lines and how they were employed. A total of 39 research papers have been reviewed from an initial set of 1388 papers. The results show that 25% of the papers reported evaluations at the Design phase of the Domain Engineering phase. The most widely used mechanism for modeling quality attributes was extended feature models and the most evaluated artifact was the base architecture. In addition, the results of the review have identified several research gaps. Specifically, 77% of the papers employed case studies as a "proof of concept" whereas 23% of the papers did not perform any type of validation. Our results are particularly relevant in positioning new research activities and in the selection of quality evaluation methods or techniques that best fit a given purpose.

[1]  Jens Knodel,et al.  Comparing Costs and Benefits of Different Test Strategies for a Software Product Line: A Study from Testo AG , 2007, 11th International Software Product Line Conference (SPLC 2007).

[2]  Stan Jarzabek,et al.  Quality Prediction and Assessment for Product Lines , 2003, CAiSE.

[3]  Antonio Ruiz Cortés,et al.  A First Approach to Build Product Lines of Multi-organizational Web Based Systems (MOWS) , 2004, IICS.

[4]  Robyn R. Lutz,et al.  Analysis of a software product line architecture: an experience report , 2003, J. Syst. Softw..

[5]  Barry W. Boehm,et al.  A software product line life cycle cost estimation model , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[6]  Emilia Mendes,et al.  A systematic review of Web engineering research , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[7]  M. Meisinger,et al.  From Scenarios to Aspects: Exploring Product Lines , 2005 .

[8]  D. Needham,et al.  A Software Fault Tree Metric , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[9]  Vojislav B. Misic,et al.  A holistic architecture assessment method for software product lines , 2007, Inf. Softw. Technol..

[10]  Nenad Medvidovic,et al.  Using service utilization metrics to assess the structure of product line architectures , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[11]  Klaus Pohl,et al.  Considering Variabilities during Component Selection in Product Family Development , 2001, PFE.

[12]  Jan Bosch,et al.  Architecting product diversification - formalizing variability dependencies in software product family engineering , 2004, Fourth International Conference onQuality Software, 2004. QSIC 2004. Proceedings..

[13]  Goiuria Sagardui Mendieta,et al.  Quality Assessment in Software Product Lines , 2008, ICSR.

[14]  Frank van der Linden,et al.  Software product lines in action , 2007 .

[15]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[16]  Goiuria Sagardui Mendieta,et al.  Quality aware software product line engineering , 2010, Journal of the Brazilian Computer Society.

[17]  Birgit Geppert,et al.  Goal-oriented assessment of product-line domains , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[18]  Muhammad Ali Babar.  Evaluating Product Line Architectures: Methods and Techniques , 2007, 14th Asia-Pacific Software Engineering Conference (APSEC'07).

[19]  Sergio Segura,et al.  FAMA: Tooling a Framework for the Automated Analysis of Feature Models , 2007, VaMoS.

[20]  Gopalakrishna Raghavan,et al.  Improving Software Quality in Product Families through Systematic Reengineering , 2002, ECSQ.

[21]  Shari Lawrence Pfleeger,et al.  Towards a Framework for Software Measurement Validation , 1995, IEEE Trans. Software Eng..

[22]  Takuya Katayama,et al.  Architectural design for evolution by analyzing requirements on quality attributes , 2001, Proceedings Eighth Asia-Pacific Software Engineering Conference.

[23]  Natalia Juristo Juzgado,et al.  Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[24]  Jens Knodel,et al.  Experiences with product line development of embedded systems at Testo AG , 2006, 10th International Software Product Line Conference (SPLC'06).

[25]  D. M. Needham,et al.  A software fault tree key node metric , 2007, J. Syst. Softw..

[26]  Christian Del Rosso.  The process of and the lessons learned from performance tuning of a product family software architecture for mobile phones , 2004, Eighth European Conference on Software Maintenance and Reengineering, 2004. CSMR 2004. Proceedings..

[27]  Eila Niemelä,et al.  Capturing quality requirements of product family architecture , 2007, Inf. Softw. Technol..

[28]  Dewayne E. Perry,et al.  Predicting Emergent Properties of Component Based Systems , 2007, 2007 Sixth International IEEE Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems (ICCBSS'07).

[29]  Silvio Romero de Lemos Meira,et al.  Evaluating Domain Design Approaches Using Systematic Review , 2008, ECSA.

[30]  Lem O. Ejiogu.  Software engineering with formal metrics , 1991 .

[31]  Klaus Schmid,et al.  Software product lines in action - the best industrial practice in product line engineering , 2007 .

[32]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[33]  Vojislav B. Misic.  Measuring the Coherence of Software Product Line Architectures , 2003, Software Engineering Research and Practice.

[34]  Felix Lösch,et al.  Optimization of Variability in Software Product Lines , 2007, 11th International Software Product Line Conference (SPLC 2007).

[35]  Robyn R. Lutz,et al.  An approach to architectural analysis of product lines , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[36]  Rafael Capilla,et al.  Modelling Variability with Features in Distributed Architectures , 2001, PFE.

[37]  Joachim Bayer,et al.  Design for Quality , 2003, PFE.

[38]  Tony Gorschek,et al.  Systematic Review of Solutions Proposed for Product Line Economics , 2008, SPLC.

[39]  Zhisheng Huang,et al.  Unified approach for system-level generative programming , 2002, Fifth International Conference on Algorithms and Architectures for Parallel Processing, 2002. Proceedings..

[40]  Sungwon Kang,et al.  Extending ATAM to assess product line architecture , 2008, 2008 8th IEEE International Conference on Computer and Information Technology.

[41]  Jan Bosch,et al.  Design and use of software architectures - adopting and evolving a product-line approach , 2000 .

[42]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[43]  Georg Herzwurm,et al.  QFD-PPP: Product Line Portfolio Planning Using Quality Function Deployment , 2005, SPLC.

[44]  Soo Dong Kim,et al.  A framework for evaluating reusability of core asset in product line engineering , 2007, Inf. Softw. Technol..

[45]  Josh Dehlinger,et al.  PLFaultCAT: A Product-Line Software Fault Tree Analysis Tool , 2006, Automated Software Engineering.

[46]  Jens Knodel,et al.  Asset recovery and their incorporation into product lines , 2005, 12th Working Conference on Reverse Engineering (WCRE'05).

[47]  Osaretin Edwin Odia.  Testing in Software Product Lines , 2007 .

[48]  Mario Piattini,et al.  Security Requirements Variability for Software Product Lines , 2008, 2008 Third International Conference on Availability, Reliability and Security.

[49]  Marco Sinnema,et al.  Using variability modeling principles to capture architectural knowledge , 2006, SOEN.

[50]  Nenad Medvidovic,et al.  Measuring Product Line Architectures , 2001, PFE.

[51]  Eila Niemelä,et al.  Quality-driven architecture design and quality analysis method: A revolutionary initiation approach to a product line architecture , 2002 .

[52]  NiemeläEila,et al.  Capturing quality requirements of product family architecture , 2007 .

[53]  Nicolas Guelfi,et al.  A Product Line Perspective for Quality Reuse of Development Frameworks for Distributed Transactional Applications , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[54]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[55]  Steffen Thiel.  On the Definition of a Framework for an Architecting Process Supporting Product Family Development , 2001, PFE.

[56]  Juha Kuusela,et al.  Consistency management of product line requirements , 2001, Proceedings Fifth IEEE International Symposium on Requirements Engineering.

[57]  Banu Diri,et al.  A Conceptual Framework to Integrate Fault Prediction Sub-Process for Software Product Lines , 2008, 2008 2nd IFIP/IEEE International Symposium on Theoretical Aspects of Software Engineering.

[58]  Miguel Toro,et al.  Automated error analysis for the agilization of feature modeling , 2008, J. Syst. Softw..