A classification of product sampling for software product lines

The analysis of software product lines is challenging due to the potentially large number of products, which grow exponentially in terms of the number of features. Product sampling is a technique used to avoid exhaustive testing, which is often infeasible. In this paper, we propose a classification for product sampling techniques and classify the existing literature accordingly. We distinguish the important characteristics of such approaches based on the information used for sampling, the kind of algorithm, and the achieved coverage criteria. Furthermore, we give an overview on existing tools and evaluations of product sampling techniques. We share our insights on the state-of-the-art of product sampling and discuss potential future work.

[1]  Pierre-Yves Schobbens,et al.  Towards an incremental automata-based approach for software product-line model checking , 2012, SPLC '12.

[2]  Miroslav Bures,et al.  Constrained Interaction Testing: A Systematic Literature Study , 2017, IEEE Access.

[3]  Silvia Regina Vergilio,et al.  A multi-objective test data generation approach for mutation testing of feature models , 2016, Journal of Software Engineering Research and Development.

[4]  Jacques Klein,et al.  Pairwise testing for software product lines: comparison of two approaches , 2012, Software Quality Journal.

[5]  Sebastian Krieter,et al.  IncLing: efficient product-line testing using incremental pairwise sampling , 2016, GPCE.

[6]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[7]  Gunter Saake,et al.  Effective product-line testing using similarity-based product prioritization , 2016, Software & Systems Modeling.

[8]  Aurora Trinidad Ramirez Pozo,et al.  Product selection based on upper confidence bound MOEA/D-DRA for testing software product lines , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[9]  Alexander Egyed,et al.  Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of Software Product Lines , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[10]  Dragan Gasevic,et al.  Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models , 2011, 2011 Eighth International Conference on Information Technology: New Generations.

[11]  Sven Apel,et al.  Family-based performance measurement , 2014 .

[12]  Aurora Trinidad Ramirez Pozo,et al.  Hyper-Heuristic Based Product Selection for Software Product Line Testing , 2017, IEEE Computational Intelligence Magazine.

[13]  Sven Apel,et al.  Scalable analysis of variable software , 2013, ESEC/FSE 2013.

[14]  Donald D. Cowan,et al.  S.P.L.O.T.: software product lines online tools , 2009, OOPSLA Companion.

[15]  Alexander Egyed,et al.  Using feature model knowledge to speed up the generation of covering arrays , 2013, VaMoS.

[16]  Sven Apel,et al.  Toward variability-aware testing , 2012, FOSD '12.

[17]  Sandro Schulze,et al.  Towards efficient SPL testing by variant reduction , 2013, VariComp@AOSD.

[18]  Malte Lochau,et al.  Pairwise feature-interaction testing for SPLs: potentials and limitations , 2011, SPLC '11.

[19]  Malte Lochau,et al.  DeltaCCS: A Core Calculus for Behavioral Change , 2014, ISoLA.

[20]  Sebastian Oster,et al.  Automated Incremental Pairwise Testing of Software Product Lines , 2010, SPLC.

[21]  Mustafa Al-Hajjaji,et al.  Optimizing product orders using graph algorithms for improving incremental product-line analysis , 2017, VaMoS.

[22]  Sven Apel,et al.  Performance-influence models for highly configurable systems , 2015, ESEC/SIGSOFT FSE.

[23]  Thomas Thüm,et al.  Applying Incremental Model Slicing to Product-Line Regression Testing , 2016, ICSR.

[24]  Gunter Saake,et al.  On essential configuration complexity: Measuring interactions in highly-configurable systems , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[25]  Kyo Chul Kang,et al.  Feature-Oriented Domain Analysis (FODA) Feasibility Study , 1990 .

[26]  Sebastian Krieter,et al.  FeatureIDE: Empowering Third-Party Developers , 2017, SPLC.

[27]  Vladimir Klebanov,et al.  Verification of Software Product Lines with Delta-Oriented Slicing , 2010, FoVeOOS.

[28]  Sven Apel,et al.  Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[29]  Jürgen Teich,et al.  Experiments on Optimizing the Performance of Stencil Codes with SPL Conqueror , 2014, Parallel Process. Lett..

[30]  Hung Viet Nguyen,et al.  Exploring variability-aware execution for testing plugin-based web applications , 2014, ICSE.

[31]  Malte Lochau,et al.  Multi-objective Test Suite Optimization for Incremental Product Family Testing , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[32]  Gunter Saake,et al.  A Classification and Survey of Analysis Strategies for Software Product Lines , 2014, ACM Comput. Surv..

[33]  Pierre-Yves Schobbens,et al.  Covering SPL Behaviour with Sampled Configurations: An Initial Assessment , 2015, VaMoS.

[34]  Yves Le Traon,et al.  Mutation-Based Generation of Software Product Line Test Configurations , 2014, SSBSE.

[35]  Ronald L. Rivest,et al.  Introduction to Algorithms, third edition , 2009 .

[36]  Malte Lochau,et al.  Incremental Model-Based Testing of Delta-Oriented Software Product Lines , 2012, TAP@TOOLS.

[37]  Thomas Leich,et al.  Mastering Software Variability with FeatureIDE , 2017, Springer International Publishing.

[38]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[39]  Sven Apel,et al.  Strategies for product-line verification: Case studies and experiments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[40]  Sven Apel,et al.  A Comparison of 10 Sampling Algorithms for Configurable Systems , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[41]  Daniel Lohmann,et al.  Understanding linux feature distribution , 2012, MISS '12.

[42]  Myra B. Cohen,et al.  Integration Testing of Software Product Lines Using Compositional Symbolic Execution , 2012, FASE.

[43]  Sven Apel,et al.  Performance‐influence models of multigrid methods: A case study on triangular grids , 2017, Concurr. Comput. Pract. Exp..

[44]  Sven Apel,et al.  The PLA model: on the combination of product-line analyses , 2013, VaMoS '13.

[45]  Jacques Klein,et al.  Multi-objective test generation for software product lines , 2013, SPLC '13.

[46]  Jackson A. Prado Lima,et al.  Automatic Generation of Search-Based Algorithms Applied to the Feature Testing of Software Product Lines , 2017, SBES'17.

[47]  Gunter Saake,et al.  Feature-Oriented Software Product Lines , 2013, Springer Berlin Heidelberg.

[48]  Sven Apel,et al.  Scalable Prediction of Non-functional Properties in Software Product Lines , 2011, 2011 15th International Software Product Line Conference.

[49]  Sergio Segura,et al.  Automated analysis of feature models 20 years later: A literature review , 2010, Inf. Syst..

[50]  Mathieu Acher,et al.  Test them all, is it worth it? A ground truth comparison of configuration sampling strategies , 2017, ArXiv.

[51]  Sarfraz Khurshid,et al.  Reducing combinatorics in testing product lines , 2011, AOSD '11.

[52]  Øystein Haugen,et al.  Generating better partial covering arrays by modeling weights on sub-product lines , 2012, MODELS'12.

[53]  Gunter Saake,et al.  Predicting performance via automated feature-interaction detection , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[54]  Uwe Aßmann,et al.  Variability-aware safety analysis using delta component fault diagrams , 2013, SPLC '13 Workshops.

[55]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[56]  Øystein Haugen,et al.  A Survey of Empirics of Strategies for Software Product Line Testing , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[57]  Anastasia Cmyrev,et al.  Efficient and Effective Testing of Automotive Software Product Lines , 2014 .

[58]  Sven Apel,et al.  Data-efficient performance learning for configurable systems , 2018, Empirical Software Engineering.

[59]  Øystein Haugen,et al.  An algorithm for generating t-wise covering arrays from large feature models , 2012, SPLC '12.

[60]  Sven Apel,et al.  Variability encoding: From compile-time to load-time variability , 2016, J. Log. Algebraic Methods Program..

[61]  Christian Dietrich,et al.  Configuration coverage in the analysis of large-scale system software , 2011, PLOS '11.

[62]  Sebastian Krieter,et al.  Tool demo: testing configurable systems with FeatureIDE , 2016, GPCE.

[63]  Sarfraz Khurshid,et al.  Reducing Configurations to Monitor in a Software Product Line , 2010, RV.

[64]  John D. McGregor,et al.  Strategies for testing products in software product lines , 2012, ACM SIGSOFT Softw. Eng. Notes.

[65]  Don S. Batory,et al.  Feature Models, Grammars, and Propositional Formulas , 2005, SPLC.

[66]  Alexander Egyed,et al.  A first systematic mapping study on combinatorial interaction testing for software product lines , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[67]  Iago Abal,et al.  Variability Bugs in Highly Configurable Systems , 2018, ACM Trans. Softw. Eng. Methodol..

[68]  Wolfgang Schröder-Preikschat,et al.  Static Analysis of Variability in System Software: The 90, 000 #ifdefs Issue , 2014, USENIX Annual Technical Conference.

[69]  Gunter Saake,et al.  Delta-oriented product prioritization for similarity-based product-line testing , 2017 .

[70]  Jacques Klein,et al.  Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines , 2014, IEEE Transactions on Software Engineering.

[71]  Tim Menzies,et al.  Scalable product line configuration: A straw to break the camel's back , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[72]  Gunter Saake,et al.  Feature-Oriented Software Product Lines , 2013, Springer Berlin Heidelberg.

[73]  SenSagar,et al.  Pairwise testing for software product lines , 2012 .

[74]  John D. McGregor,et al.  On strategies for testing software product lines: A systematic literature review , 2014, Inf. Softw. Technol..

[75]  Sungwon Kang,et al.  A survey on software product line testing , 2012, SPLC '12.

[76]  ApelSven,et al.  A Classification and Survey of Analysis Strategies for Software Product Lines , 2014 .

[77]  Paolo Arcaini,et al.  Generating Tests for Detecting Faults in Feature Models , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[78]  Dragan Gasevic,et al.  Evolutionary Search-Based Test Generation for Software Product Line Feature Models , 2012, CAiSE.

[79]  Alexander Egyed,et al.  A preliminary empirical assessment of similarity for combinatorial interaction testing of software product lines , 2016, SBST@ICSE.

[80]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[81]  Øystein Haugen,et al.  Properties of realistic feature models make combinatorial testing of product lines feasible , 2011, MODELS'11.

[82]  Arnaud Gotlieb,et al.  Practical pairwise testing for software product lines , 2013, SPLC '13.

[83]  Gunter Saake,et al.  SPL Conqueror: Toward optimization of non-functional properties in software product lines , 2012, Software Quality Journal.

[84]  Mario Piattini,et al.  Systematic Review on Software Product Line Testing , 2010, ICSOFT.

[85]  Udo Kelter,et al.  Fault-based product-line testing: effective sample generation based on feature-diagram mutation , 2015, SPLC.

[86]  Claus Brabrand,et al.  Systematic derivation of static analyses for software product lines , 2014, MODULARITY.