Potential Errors and Test Assessment in Software Product Line Engineering

Software product lines (SPL) are a method for the development of variant-rich software systems. Compared to non-variable systems, testing SPLs is extensive due to an increasingly amount of possible products. Different approaches exist for testing SPLs, but there is less research for assessing the quality of these tests by means of error detection capability. Such test assessment is based on error injection into correct version of the system under test. However to our knowledge, potential errors in SPL engineering have never been systematically identified before. This article presents an overview over existing paradigms for specifying software product lines and the errors that can occur during the respective specification processes. For assessment of test quality, we leverage mutation testing techniques to SPL engineering and implement the identified errors as mutation operators. This allows us to run existing tests against defective products for the purpose of test assessment. From the results, we draw conclusions about the error-proneness of the surveyed SPL design paradigms and how quality of SPL tests can be improved.

[1]  Hassan Gomaa,et al.  Model-based testing for applications derived from software product lines , 2005, A-MOST.

[2]  Richard A DeMillo.  Mutation Analysis as a Tool for Software Quality Assurance. , 1980 .

[3]  Bernhard Rumpe,et al.  Modeling Variants of Automotive Systems using Views , 2014, ArXiv.

[4]  Jin-hua Li,et al.  Mutation Analysis for Testing Finite State Machines , 2009, 2009 Second International Symposium on Electronic Commerce and Security.

[5]  Hassan Gomaa,et al.  Model-based testing for applications derived from software product lines , 2005, ACM SIGSOFT Softw. Eng. Notes.

[6]  J. McDermid,et al.  Test Data Generation for Product Lines – A Mutation Testing Approach , 2004 .

[7]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[8]  M. R. Woodward,et al.  Errors in algebraic specifications and an experimental mutation testing tool , 1993, Softw. Eng. J..

[9]  Florian Lorber,et al.  Time for Mutants - Model-Based Mutation Testing with Timed Automata , 2013, TAP@STAF.

[10]  Stephan Weißleder,et al.  Influencing Factors in Model-Based Testing with UML State Machines: Report on an Industrial Cooperation , 2009, MoDELS.

[11]  Klaus Pohl,et al.  Software Product Line Engineering - Foundations, Principles, and Techniques , 2005 .

[12]  Kyo Chul Kang,et al.  Feature-Oriented Domain Analysis (FODA) Feasibility Study , 1990 .

[13]  Fevzi Belli,et al.  Event‐based modelling, analysis and testing of user interactions: approach and case study , 2006, Softw. Test. Verification Reliab..

[14]  Malte Lochau,et al.  MoSo-PoLiTe: tool support for pairwise and model-based software product line testing , 2011, VaMoS '11.

[15]  Fevzi Belli,et al.  Test generation and minimization with "basic" statecharts , 2008, SAC '08.

[16]  Markus Scheidgen,et al.  Refactorings in Language Development with Asymmetric Bidirectional Model Transformations , 2013, SDL Forum.

[17]  John A. Clark,et al.  Assessing Test Set Adequacy for Object-Oriented Programs Using Class Mutation , 2000 .

[18]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[19]  John A. Clark,et al.  Class Mutation : Mutation Testing for Object-Oriented Programs , 2000 .

[20]  Thomas S. Heinze,et al.  Efficient Test Suite Reduction by Merging Pairs of Suitable Test Cases , 2010, MoDELS Workshops.

[21]  Thomas S. Heinze,et al.  Efficient Reduction of Model-Based Generated Test Suites through Test Case Pair Prioritization , 2010, 2010 Workshop on Model-Driven Engineering, Verification, and Validation.

[22]  Ina Schaefer,et al.  Variability Modelling for Model-Driven Development of Software Product Lines , 2010, VaMoS.

[23]  Markus Voelter,et al.  Expressing Feature-Based Variability in Structural Models , 2007 .

[24]  Shaoying Liu,et al.  Generating test data from state‐based specifications , 2003, Softw. Test. Verification Reliab..

[25]  Andy Schürr,et al.  Reduktion von Testsuiten für Software-Produktlinien , 2012, Software Engineering.

[26]  Fevzi Belli,et al.  Model-Based Integration Testing with Communication Sequence Graphs , 2011, Model-Based Testing for Embedded Systems.

[27]  Klaus Pohl,et al.  Software Product Line Engineering , 2005 .

[28]  Vadim Okun,et al.  Mutation operators for specifications , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[29]  Malte Lochau,et al.  Incremental Model-Based Testing of Delta-Oriented Software Product Lines , 2012, TAP@TOOLS.

[30]  Paul Clements,et al.  Software product lines - practices and patterns , 2001, SEI series in software engineering.

[31]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[32]  Florian Wartenberg,et al.  Model-Based Test Design of Product Lines: Raising Test Design to the Product Line Level , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[33]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[34]  Jacques Klein,et al.  Assessing Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[35]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[36]  Michal Antkiewicz,et al.  Mapping features to models: a template approach based on superimposed variants , 2005, GPCE'05.