Localizing configurations in highly-configurable systems

The complexity of configurable systems has grown immensely, and it is only getting more complex. Such systems are a challenge for software testing and maintenance, because bugs and other defects can and do appear in any configuration. One common requirement for many development tasks is to identify the configurations that lead to a given defect or some other program behavior. We distill this requirement down to a challenge question: given a program location in a source file, what are valid configurations that include the location? The key obstacle is scalability. When there are thousands of configuration options, enumerating all combinations is exponential and infeasible. We provide a set of target programs of increasing difficulty and variations on the challenge question so that submitters of all experience levels can try out solutions. Our hope is to engage the community and stimulate new and interesting approaches to the problem of analyzing configurations.

[1]  Sebastian Erdweg,et al.  Variability-aware parsing in the presence of lexical macros and conditional compilation , 2011, OOPSLA '11.

[2]  Paul Gazzillo,et al.  Kmax: finding all configurations of Kbuild makefiles statically , 2017, ESEC/SIGSOFT FSE.

[3]  Richard C. Holt,et al.  Mining Kbuild to Detect Variability Anomalies in Linux , 2012, 2012 16th European Conference on Software Maintenance and Reengineering.

[4]  Yu Lei,et al.  Advanced Combinatorial Test Methods for System Reliability , 2010 .

[5]  Wolfgang De Meuter,et al.  Design recovery and maintenance of build systems , 2007, 2007 IEEE International Conference on Software Maintenance.

[6]  Krzysztof Czarnecki,et al.  Feature-to-Code Mapping in Two Large Product Lines , 2010, SPLC.

[7]  Adam A. Porter,et al.  iTree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees , 2014, IEEE Transactions on Software Engineering.

[8]  Sven Apel,et al.  Presence-Condition Simplification in Highly Configurable Systems , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[9]  Krzysztof Czarnecki,et al.  Verifying feature-based model templates against well-formedness OCL constraints , 2006, GPCE '06.

[10]  Myra B. Cohen,et al.  GUI Interaction Testing: Incorporating Event Context , 2011, IEEE Transactions on Software Engineering.

[11]  Eric Bodden,et al.  Tracking Load-Time Configuration Options , 2014, IEEE Transactions on Software Engineering.

[12]  Sven Apel,et al.  A Comparison of 10 Sampling Algorithms for Configurable Systems , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[13]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[14]  Sven Apel,et al.  Exploring feature interactions in the wild: the new feature-interaction challenge , 2013, FOSD '13.

[15]  Ralph E. Johnson,et al.  Analyzing multiple configurations of a C program , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[16]  Adam A. Porter,et al.  Using symbolic evaluation to understand behavior in configurable software systems , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[17]  Myra B. Cohen,et al.  Moving Forward with Combinatorial Interaction Testing , 2014, Computer.

[18]  Michael D. Ernst,et al.  Which configuration option should I change? , 2014, ICSE.

[19]  Hung Viet Nguyen,et al.  Build code analysis with symbolic evaluation , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[20]  Krzysztof Czarnecki,et al.  Where Do Configuration Constraints Stem From? An Extraction Approach and an Empirical Study , 2015, IEEE Transactions on Software Engineering.

[21]  Wolfgang Schröder-Preikschat,et al.  A robust approach for variability extraction from the Linux build system , 2012, SPLC '12.

[22]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[23]  Gunter Saake,et al.  Analysis Strategies for Software Product Lines , 2012 .

[24]  Adam A. Porter,et al.  iGen: dynamic interaction inference for configurable software , 2016, SIGSOFT FSE.

[25]  Ettore Merlo,et al.  Locating features in dynamically configured avionics software , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[26]  Myra B. Cohen,et al.  Beyond the rainbow: self-adaptive failure avoidance in configurable systems , 2014, SIGSOFT FSE.

[27]  Robert Grimm,et al.  SuperC: parsing all of C by taming the preprocessor , 2012, PLDI.

[28]  Don S. Batory,et al.  Finding near-optimal configurations in product lines by random sampling , 2017, ESEC/SIGSOFT FSE.

[29]  R. Weisberg.  A-N-D , 2011 .

[30]  Angelo Gargantini,et al.  Code-Aware Combinatorial Interaction Testing , 2019, IET Softw..

[31]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[32]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[33]  Iago Abal,et al.  42 variability bugs in the linux kernel: a qualitative analysis , 2014, ASE.

[34]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[35]  Myra B. Cohen,et al.  Feedback driven adaptive combinatorial testing , 2011, ISSTA '11.

[36]  Stefan Sobernig,et al.  Attributed variability models: outside the comfort zone , 2017, ESEC/SIGSOFT FSE.

[37]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..