Tool demo: testing configurable systems with FeatureIDE

Most software systems are designed to provide custom functionality using configuration options. Testing such systems is challenging as running tests of a single configuration is often not sufficient, because defects may appear in other configurations. Ideally, all configurations of a software system should be tested, which is usually not applicable in practice due to the combinatorial explosion with respect to the configuration options. Multiple sampling strategies aim to reduce the set of tested configurations to a feasible amount, such as T-wise sampling, random configurations, and user-defined configurations. However, these strategies are often not applied in practice as they require manual effort or a specialized testing framework. Within our tool FeatureIDE, we integrate all aforementioned strategies and reduce the manual effort by automating the process of generating and testing configurations. Furthermore, we provide support for unit testing to avoid redundant test executions and for variability-aware testing. With this extension of FeatureIDE, we aim to make recent testing techniques for configurable systems applicable in practice.

[1]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[2]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[3]  Øystein Haugen,et al.  An algorithm for generating t-wise covering arrays from large feature models , 2012, SPLC '12.

[4]  Øystein Haugen,et al.  Properties of realistic feature models make combinatorial testing of product lines feasible , 2011, MODELS'11.

[5]  John D. McGregor,et al.  Testing a Software Product Line , 2001, PSSE.

[6]  Sven Apel,et al.  Strategies for product-line verification: Case studies and experiments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[7]  Sebastian Krieter,et al.  FeatureIDE: Scalable Product Configuration of Variable Systems , 2016, ICSR.

[8]  Gunter Saake,et al.  Potential synergies of theorem proving and model checking for software product lines , 2014, SPLC.

[9]  Márcio Ribeiro,et al.  The Love/Hate Relationship with the C Preprocessor: An Interview Study , 2015, ECOOP.

[10]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[11]  Wolfgang Schröder-Preikschat,et al.  Feature consistency in compile-time-configurable system software: facing the linux 10,000 feature problem , 2011, EuroSys '11.

[12]  Vasek Chvátal,et al.  A Greedy Heuristic for the Set-Covering Problem , 1979, Math. Oper. Res..

[13]  Gunter Saake,et al.  On essential configuration complexity: Measuring interactions in highly-configurable systems , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[14]  Bernhard Beckert,et al.  Verification of Object-Oriented Software. The KeY Approach - Foreword by K. Rustan M. Leino , 2007, The KeY Approach.

[15]  Thomas Leich,et al.  FeatureIDE: An extensible framework for feature-oriented software development , 2014, Sci. Comput. Program..

[16]  Sven Apel,et al.  Facilitating Reuse in Multi-goal Test-Suite Generation for Software Product Lines , 2015, FASE.

[17]  Malte Lochau,et al.  Industrial evaluation of pairwise SPL testing with MoSo-PoLiTe , 2012, VaMoS.

[18]  Don S. Batory,et al.  Scaling step-wise refinement , 2004, IEEE Transactions on Software Engineering.

[19]  Jacques Klein,et al.  PLEDGE: a product line editor and test generation tool , 2013, SPLC '13 Workshops.

[20]  Klaus Havelund,et al.  Model checking JAVA programs using JAVA PathFinder , 2000, International Journal on Software Tools for Technology Transfer.

[21]  Gregor Kiczales,et al.  Aspect-oriented programming , 2001, ESEC/FSE-9.

[22]  Sven Apel,et al.  Introducing Binary Decision Diagrams in the explicit-state verification of Java code , 2011 .

[23]  Sven Apel,et al.  Language-Independent and Automated Software Composition: The FeatureHouse Experience , 2013, IEEE Transactions on Software Engineering.

[24]  Sebastian Krieter,et al.  FeatureIDE: Taming the Preprocessor Wilderness , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[25]  Gunter Saake,et al.  A Classification and Survey of Analysis Strategies for Software Product Lines , 2014, ACM Comput. Surv..

[26]  Don S. Batory,et al.  Feature Models, Grammars, and Propositional Formulas , 2005, SPLC.

[27]  Gunter Saake,et al.  Feature-Oriented Software Product Lines , 2013, Springer Berlin Heidelberg.

[28]  Patrick Th. Eugster,et al.  Semantics-aware trace analysis , 2009, PLDI '09.

[29]  Daniel Le Berre,et al.  The Sat4j library, release 2.2 , 2010, J. Satisf. Boolean Model. Comput..

[30]  Xiangyu Zhang,et al.  Efficient program execution indexing , 2008, PLDI '08.

[31]  Gunter Saake,et al.  Feature-Oriented Software Product Lines , 2013, Springer Berlin Heidelberg.

[32]  Sebastian Krieter,et al.  IncLing: efficient product-line testing using incremental pairwise sampling , 2016, GPCE.

[33]  Michael Jackson,et al.  Distributed Feature Composition: A Virtual Architecture for Telecommunications Services , 1998, IEEE Trans. Software Eng..

[34]  Sarfraz Khurshid,et al.  SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems , 2013, ESEC/FSE 2013.

[35]  Gunter Saake,et al.  Similarity-based prioritization in software product-line testing , 2014, SPLC.

[36]  Sergio Segura,et al.  Automated analysis of feature models 20 years later: A literature review , 2010, Inf. Syst..