Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?

Uniform or near-uniform generation of solutions for large satisfiability formulas is a problem of theoretical and practical interest for the testing community. Recent works proposed two algorithms (namely UniGen and QuickSampler) for reaching a good compromise between execution time and uniformity guarantees, with empirical evidence on SAT benchmarks. In the context of highly-configurable software systems (e.g., Linux), it is unclear whether UniGen and QuickSampler can scale and sample uniform software configurations. In this paper, we perform a thorough experiment on 128 real-world feature models. We find that UniGen is unable to produce SAT solutions out of such feature models. Furthermore, we show that QuickSampler does not generate uniform samples and that some features are either never part of the sample or too frequently present. Finally, using a case study, we characterize the impacts of these results on the ability to find bugs in a configurable system. Overall, our results suggest that we are not there: more research is needed to explore the cost-effectiveness of uniform sampling when testing large configurable systems.

[1]  Bart Selman,et al.  Uniform Solution Sampling Using a Constraint Solver As an Oracle , 2012, UAI.

[2]  Øystein Haugen,et al.  An algorithm for generating t-wise covering arrays from large feature models , 2012, SPLC '12.

[3]  Sven Apel,et al.  A Comparison of 10 Sampling Algorithms for Configurable Systems , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[4]  Lionel C. Briand,et al.  Formal Analysis of the Probability of Interaction Fault Detection Using Random Testing , 2012, IEEE Transactions on Software Engineering.

[5]  Krzysztof Czarnecki,et al.  SAT-based analysis of feature models is easy , 2009, SPLC.

[6]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[7]  Pierre-Yves Schobbens,et al.  Feature Diagrams: A Survey and a Formal Semantics , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[8]  Mustafa Al-Hajjaji,et al.  A classification of product sampling for software product lines , 2018, SPLC.

[9]  Marc Thurley,et al.  sharpSAT - Counting Models with Advanced Component Caching and Implicit BCP , 2006, SAT.

[10]  Vibhav Gogate,et al.  A New Algorithm for Sampling CSP Solutions Uniformly at Random , 2006, CP.

[11]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[12]  Dragan Gasevic,et al.  Evolutionary Search-Based Test Generation for Software Product Line Feature Models , 2012, CAiSE.

[13]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[14]  Supratik Chakraborty,et al.  Balancing scalability and uniformity in SAT witness generator , 2014, 2014 51st ACM/EDAC/IEEE Design Automation Conference (DAC).

[15]  Sharad Malik,et al.  On computing minimal independent support and its applications to sampling and counting , 2015, Constraints.

[16]  Mathieu Acher,et al.  Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack , 2017, Empirical Software Engineering.

[17]  Bart Selman,et al.  Towards Efficient Sampling: Exploiting Random Walk Strategies , 2004, AAAI.

[18]  Sanjit A. Seshia,et al.  On Parallel Scalable Uniform SAT Witness Generation , 2015, TACAS.

[19]  Bart Selman,et al.  Embed and Project: Discrete Sampling with Universal Hashing , 2013, NIPS.

[20]  Stephan Mennicke,et al.  Is there a mismatch between real-world feature models and product-line research? , 2017, ESEC/SIGSOFT FSE.

[21]  Jacques Klein,et al.  Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines , 2014, IEEE Transactions on Software Engineering.

[22]  K. Miettinen,et al.  Quasi-random initial population for genetic algorithms , 2004 .

[23]  Myra B. Cohen,et al.  Navigating the Maze: The Impact of Configurability in Bioinformatics Software , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[24]  Krzysztof Czarnecki,et al.  SMTIBEA: a hybrid multi-objective optimization algorithm for configuring large constrained software product lines , 2019, Software & Systems Modeling.

[25]  Supratik Chakraborty,et al.  A Scalable and Nearly Uniform Generator of SAT Witnesses , 2013, CAV.

[26]  Tim Menzies,et al.  On the value of user preferences in search-based software engineering: A case study in software product lines , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[27]  Benjamin Doerr,et al.  Money for Nothing: Speeding Up Evolutionary Algorithms Through Better Initialization , 2015, GECCO.

[28]  Gunter Saake,et al.  Feature-Oriented Software Product Lines , 2013, Springer Berlin Heidelberg.

[29]  Alexander Egyed,et al.  A first systematic mapping study on combinatorial interaction testing for software product lines , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[30]  Yves Le Traon,et al.  Combining Multi-Objective Search and Constraint Solving for Configuring Large Software Product Lines , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[31]  Sebastian Krieter,et al.  Propagating Configuration Decisions with Modal Implication Graphs , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[32]  Iago Abal,et al.  42 variability bugs in the linux kernel: a qualitative analysis , 2014, ASE.

[33]  Andreas Kuehlmann,et al.  Stimulus generation for constrained random simulation , 2007, 2007 IEEE/ACM International Conference on Computer-Aided Design.

[34]  Antonio Ruiz Cortés,et al.  Automated analysis of feature models: challenges ahead , 2006, CACM.

[35]  Krzysztof Czarnecki,et al.  SAT-based analysis of large real-world feature models is easy , 2015, SPLC.

[36]  Erick Cantú-Paz,et al.  On Random Numbers and the Performance of Genetic Algorithms , 2002, GECCO.

[37]  Koushik Sen,et al.  Efficient Sampling of SAT Solutions for Testing , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[38]  Sergio Segura,et al.  Automated analysis of feature models 20 years later: A literature review , 2010, Inf. Syst..

[39]  Gunter Saake,et al.  Similarity-based prioritization in software product-line testing , 2014, SPLC.