Validation of Constraints Among Configuration Parameters Using Search-Based Combinatorial Interaction Testing

The appeal of highly-configurable software systems lies in their adaptability to users’ needs. Search-based Combinatorial Interaction Testing (CIT) techniques have been specifically developed to drive the systematic testing of such highly-configurable systems. In order to apply these, it is paramount to devise a model of parameter configurations which conforms to the software implementation. This is a non-trivial task. Therefore, we extend traditional search-based CIT by devising 4 new testing policies able to check if the model correctly identifies constraints among the various software parameters. Our experiments show that one of our new policies is able to detect faults both in the model and the software implementation that are missed by the standard approaches.

[1]  Yuanyuan Zhang,et al.  Search based software engineering for software product line engineering: a survey and directions for future work , 2014, SPLC.

[2]  Myra B. Cohen,et al.  An Improved Meta-heuristic Search for Constrained Interaction Testing , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[3]  Mathieu Acher,et al.  Feature model extraction from large collections of informal product descriptions , 2013, ESEC/FSE 2013.

[4]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[5]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[6]  Paolo Arcaini,et al.  Automatic Detection and Removal of Conformance Faults in Feature Models , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[7]  Silvia Regina Vergilio,et al.  A Mutation Approach to Feature Testing of Software Product Lines , 2013, SEKE.

[8]  Alexander Egyed,et al.  A systematic mapping study of search-based software engineering for software product lines , 2015, Inf. Softw. Technol..

[9]  Wolfgang Schröder-Preikschat,et al.  Feature consistency in compile-time-configurable system software: facing the linux 10,000 feature problem , 2011, EuroSys '11.

[10]  Jeff Yu Lei,et al.  Combinatorial Software Testing , 2009, Computer.

[11]  Eitan Farchi,et al.  Using binary decision diagrams for combinatorial test design , 2011, ISSTA '11.

[12]  Angelo Gargantini,et al.  Combinatorial Interaction Testing with CITLAB , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[13]  Myra B. Cohen,et al.  Reducing Masking Effects in CombinatorialInteraction Testing: A Feedback DrivenAdaptive Approach , 2014, IEEE Transactions on Software Engineering.

[14]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[15]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[16]  Angelo Gargantini,et al.  CITLAB: A Laboratory for Combinatorial Interaction Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[17]  Myra B. Cohen,et al.  Integration Testing of Software Product Lines Using Compositional Symbolic Execution , 2012, FASE.

[18]  Jacques Klein,et al.  Towards automated testing and fixing of re-engineered Feature Models , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[19]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[20]  Myra B. Cohen,et al.  Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection , 2015, IEEE Transactions on Software Engineering.

[21]  Angelo Gargantini,et al.  A Formal Logic Approach to Constrained Combinatorial Testing , 2010, Journal of Automated Reasoning.

[22]  Alexander Egyed,et al.  Reverse Engineering Feature Models with Evolutionary Algorithms: An Exploratory Study , 2012, SSBSE.

[23]  D. Richard Kuhn,et al.  Pseudo-Exhaustive Testing for Software , 2006, 2006 30th Annual IEEE/NASA Software Engineering Workshop.

[24]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[25]  Krzysztof Czarnecki,et al.  Mining configuration constraints: static analyses and empirical results , 2014, ICSE.