Scalable analysis of variable software

The advent of variability management and generator technology enables users to derive individual variants from a variable code base based on a selection of desired configuration options. This approach gives rise to the generation of possibly billions of variants that, however, cannot be efficiently analyzed for errors with classic analysis techniques. To address this issue, researchers and practitioners usually apply sampling heuristics. While sampling reduces the analysis effort significantly, the information obtained is necessarily incomplete and it is unknown whether sampling heuristics scale to billions of variants. Recently, researchers have begun to develop variability-aware analyses that analyze the variable code base directly exploiting the similarities among individual variants to reduce analysis effort. However, while being promising, so far, variability-aware analyses have been applied mostly only to small academic systems. To learn about the mutual strengths and weaknesses of variability-aware and sampling-based analyses of software systems, we compared the two strategies by means of two concrete analysis implementations (type checking and liveness analysis), applied them to three subject systems: Busybox, the x86 Linux kernel, and OpenSSL. Our key finding is that variability-aware analysis outperforms most sampling heuristics with respect to analysis time while preserving completeness.

[1]  Michel Dagenais,et al.  C/C++ conditional compilation analysis using symbolic execution , 2000, Proceedings 2000 International Conference on Software Maintenance.

[2]  Pierre-Yves Schobbens,et al.  Model checking lots of systems: efficient verification of temporal properties in software product lines , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[3]  Daniel Lohmann,et al.  Understanding linux feature distribution , 2012, MISS '12.

[4]  William R. Cook,et al.  Safe composition of product lines , 2007, GPCE '07.

[5]  William R. Cook,et al.  Fitting the pieces together: a machine-checked model of safe composition , 2009, ESEC/FSE '09.

[6]  Martin Erwig,et al.  An error-tolerant type system for variational lambda calculus , 2012, ICFP.

[7]  Wolfgang Schröder-Preikschat,et al.  Feature consistency in compile-time-configurable system software: facing the linux 10,000 feature problem , 2011, EuroSys '11.

[8]  Sven Apel,et al.  Family-based deductive verification of software product lines , 2012, GPCE '12.

[9]  Stephan Merz,et al.  Model Checking , 2000 .

[10]  Sven Apel,et al.  An Overview of Feature-Oriented Software Development , 2009, J. Object Technol..

[11]  Christian Dietrich,et al.  Configuration coverage in the analysis of large-scale system software , 2011, PLOS '11.

[12]  Mario Latendresse Fast symbolic evaluation of C/C++ preprocessing using conditional values , 2003, Seventh European Conference onSoftware Maintenance and Reengineering, 2003. Proceedings..

[13]  Sven Apel,et al.  Introducing Binary Decision Diagrams in the explicit-state verification of Java code , 2011 .

[14]  Lerina Aversano,et al.  Handling preprocessor-conditioned declarations , 2002, Proceedings. Second IEEE International Workshop on Source Code Analysis and Manipulation.

[15]  Øystein Haugen,et al.  An algorithm for generating t-wise covering arrays from large feature models , 2012, SPLC '12.

[16]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[17]  Muffy Calder,et al.  Feature interaction detection by pairwise analysis of LTL properties—A case study , 2006, Formal Methods Syst. Des..

[18]  Mira Mezini,et al.  SPLLIFT: statically analyzing software product lines in minutes instead of years , 2013, Software Engineering.

[19]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[20]  Krzysztof Czarnecki,et al.  Generative programming - methods, tools and applications , 2000 .

[21]  Helmut Veith,et al.  Counterexample-guided abstraction refinement for symbolic model checking , 2003, JACM.

[22]  Krzysztof Czarnecki,et al.  Variability modeling in the real: a perspective from the operating systems domain , 2010, ASE '10.

[23]  Øystein Haugen,et al.  Properties of realistic feature models make combinatorial testing of product lines feasible , 2011, MODELS'11.

[24]  Klaus Pohl,et al.  Model Checking of Domain Artifacts in Product Line Engineering , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[25]  Wolfgang Schröder-Preikschat,et al.  Is The Linux Kernel a Software Product Line , 2007 .

[26]  Sven Apel,et al.  Type safety for feature-oriented product lines , 2010, Automated Software Engineering.

[27]  Gunter Saake,et al.  Analysis Strategies for Software Product Lines , 2012 .

[28]  Sven Apel,et al.  Strategies for product-line verification: Case studies and experiments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[29]  Sven Apel,et al.  Detection of feature interactions using feature-aware verification , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[30]  Martin Erwig,et al.  The Choice Calculus: A Representation for Software Variation , 2011, TSEM.

[31]  Sven Apel,et al.  Analyzing the discipline of preprocessor annotations in 30 million lines of C code , 2011, AOSD '11.

[32]  Gunter Saake,et al.  Predicting performance via automated feature-interaction detection , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[33]  Masaru Tomita,et al.  LR Parsers For Natural Languages , 1984, Annual Meeting of the Association for Computational Linguistics.

[34]  Jacques Klein,et al.  Pairwise testing for software product lines: comparison of two approaches , 2012, Software Quality Journal.

[35]  Sebastian Erdweg,et al.  A variability-aware module system , 2012, OOPSLA '12.

[36]  Sven Apel,et al.  Granularity in software product lines , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[37]  Gunter Saake,et al.  Type checking annotation-based product lines , 2012, TSEM.

[38]  Krzysztof Czarnecki,et al.  SAT-based analysis of feature models is easy , 2009, SPLC.

[39]  Myra B. Cohen,et al.  Feature Interaction Faults Revisited: An Exploratory Study , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[40]  Malte LochauSebastian Model-based pairwise testing for feature interaction coverage in software product line engineering , 2012 .

[41]  Sven Apel,et al.  Toward variability-aware testing , 2012, FOSD '12.

[42]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[43]  Krzysztof Czarnecki,et al.  Feature-to-Code Mapping in Two Large Product Lines , 2010, SPLC.

[44]  Sebastian Erdweg,et al.  Variability-aware parsing in the presence of lexical macros and conditional compilation , 2011, OOPSLA '11.

[45]  Sven Apel,et al.  An analysis of the variability in forty preprocessor-based software product lines , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[46]  Sebastian Oster,et al.  Automated Incremental Pairwise Testing of Software Product Lines , 2010, SPLC.

[47]  Claus Brabrand,et al.  Intraprocedural dataflow analysis for software product lines , 2012, AOSD.