Strategies for testing products in software product lines

The software product line engineering strategy enables the achievement of significant improvements in quality through reuse of carefully crafted software assets across multiple products. However, high levels of quality in the software product line assets, which are used to create products, must be accompanied by effective and efficient test strategies for the products in the software product line. The goal of this study is to understand which strategies for testing products in software product lines have been reported in the literature, enabling discussions on the significant issues, and also pointing out further research directions. A systematic literature review was carried out that identified two hundred seventy-three papers, published from the years 1998 and early in 2012. From such a set of papers, a systematic selection resulted in forty-one relevant papers. The analysis of the reported strategies comprised two important aspects: the selection of products for testing, and the actual test of products. The findings showed a range of strategies, dealing with both aspects, but few empirical evaluations of their effectiveness have been performed, which limits the inferences that can be drawn.

[1]  Hassan Gomaa,et al.  Reusable Model-Based Testing , 2009, ICSR.

[2]  Sarfraz Khurshid,et al.  Incremental Test Generation for Software Product Lines , 2010, IEEE Transactions on Software Engineering.

[3]  Clémentine Nebut,et al.  Automated requirements-based generation of test cases for product families , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[4]  Andy Schürr,et al.  Towards Software Product Line Testing using Story Driven Modelling , 2008 .

[5]  Macario Polo,et al.  Automatic Generation of Test Cases in Software Product Lines , 2009, ICEIS.

[6]  Paul G. Sorenson,et al.  Testing Software Assets of Framework-Based Product Families During Application Engineering Stage , 2008, J. Softw..

[7]  Sarfraz Khurshid,et al.  Reducing combinatorics in testing product lines , 2011, AOSD '11.

[8]  Mario Piattini,et al.  A Model Based Testing Approach for Model-Driven Development and Software Product Lines , 2010, ENASE.

[9]  Sebastian Oster,et al.  Automated Incremental Pairwise Testing of Software Product Lines , 2010, SPLC.

[10]  Ramadan Moawad,et al.  An approach for requirements based software product line testing , 2010, 2010 The 7th International Conference on Informatics and Systems (INFOS).

[11]  Jaejoon Lee,et al.  Concepts and Guidelines of Feature Modeling for Product Line Software Engineering , 2002, ICSR.

[12]  Sergio Segura,et al.  Automated metamorphic testing on the analyses of feature models , 2011, Inf. Softw. Technol..

[13]  Klaus Pohl,et al.  A Reuse Technique for Performance Testing of Software Product Lines , 2006 .

[14]  Per Runeson,et al.  Empirical evaluations of regression test selection techniques: a systematic review , 2008, ESEM '08.

[15]  Dragan Gasevic,et al.  Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models , 2011, 2011 Eighth International Conference on Information Technology: New Generations.

[16]  Andy Schürr,et al.  Model-based coverage-driven test suite generation for software product lines , 2011, MODELS'11.

[17]  Hassan Gomaa,et al.  Model-based testing for applications derived from software product lines , 2005, ACM SIGSOFT Softw. Eng. Notes.

[18]  Matthias Grochtmann,et al.  Modelling Requirements to Support Testing of Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[19]  Klaus Pohl,et al.  Avoiding Redundant Testing in Application Engineering , 2010, SPLC.

[20]  K. D. Scheidemann.  Optimizing the selection of representative configurations in verification of evolving product lines of distributed embedded systems , 2006 .

[21]  Jeff Yu Lei,et al.  Combinatorial Testing of ACTS: A Case Study , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[22]  Juha Taina,et al.  Product family testing: a survey , 2004, SOEN.

[23]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[24]  Andreas Reuys,et al.  Derivation of Domain Test Scenarios from Activity Diagrams1 , 2003 .

[25]  Myra B. Cohen,et al.  Integration Testing of Software Product Lines Using Compositional Symbolic Execution , 2012, FASE.

[26]  Paul Clements,et al.  Better, faster, cheaper: Pick any three , 2012 .

[27]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[28]  Arnaud Gotlieb,et al.  PACOGEN: Automatic Generation of Pairwise Test Configurations from Feature Models , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[29]  Myra B. Cohen,et al.  Improving the Testing and Testability of Software Product Lines , 2010, SPLC.

[30]  Klaus Pohl,et al.  The ScenTED Method for Testing Software Product Lines , 2006, Software Product Lines.

[31]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[32]  Øystein Haugen,et al.  Properties of realistic feature models make combinatorial testing of product lines feasible , 2011, MODELS'11.

[33]  Jean-Marc Jézéquel,et al.  Reusable test requirements for UML-modeled product lines , 2002 .

[34]  Satish Mishra.  Specification Based Software Product Line Testing : A case study , 2006 .

[35]  Markus Roggenbach,et al.  Speci?cation-Based Testing for Software Product Lines , 2008, 2008 Sixth IEEE International Conference on Software Engineering and Formal Methods.

[36]  Klaus Pohl,et al.  Testing Variabilities in Use Case Models , 2003, PFE.

[37]  Clémentine Nebut,et al.  System Testing of Product Lines: From Requirements to Test Cases , 2006, Software Product Lines.

[38]  Thomas Gustafsson.  An Approach for Selecting Software Product Line Instances for Testing , 2007, SPLC.

[39]  Macario Polo,et al.  Testing Product Generation in Software Product Lines Using Pairwise for Features Coverage , 2010, ICTSS.

[40]  Macario Polo,et al.  Model-driven testing in software product lines , 2009, 2009 IEEE International Conference on Software Maintenance.

[41]  Mario Piattini,et al.  Software Product Line Testing - A Systematic Review , 2009, ICSOFT.

[42]  Jesús García-Galán,et al.  BETTY: Benchmarking and Testing on the Automated Analysis of Feature Models , 2012 .

[43]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[44]  Malte LochauSebastian.  Model-based pairwise testing for feature interaction coverage in software product line engineering , 2012 .

[45]  Malte Lochau,et al.  Pairwise feature-interaction testing for SPLs: potentials and limitations , 2011, SPLC '11.

[46]  Birgit Geppert,et al.  Enabling Verifiable Conformance for Product Lines , 2008, 2008 12th International Software Product Line Conference.

[47]  John D. McGregor,et al.  A systematic mapping study of software product lines testing , 2011, Inf. Softw. Technol..

[48]  John D. McGregor,et al.  Testing a Software Product Line , 2001, PSSE.

[49]  Antonia Bertolino,et al.  Product Line Use Cases: Scenario-Based Specification and Testing of Requirements , 2006, Software Product Lines.