Search-based product line fault detection allocating test cases iteratively

The large number of possible configurations makes it unfeasible to test every single system variant in a product line. Consequently, a small subset of the product line products must be selected, typically following combinatorial interaction testing approaches. Recently, many product line engineering approaches have considered the selection and prioritization of relevant products within the product line. In a further step, these products are thoroughly tested individually. However, the test cases that must be executed in each of the products are not always insignificant, and in systems such as Cyber-Physical System Product Lines (CPSPLs), their test execution time can vary from tens to thousands of seconds. This issue leads to spending a lot of time testing each individual product. To solve this problem we propose a search-based approach to perform the testing of product lines by allocating small number of test cases in each of the products. This approach increases the probability of detecting faults faster. Specifically, our search-based approach obtains a set of products, which are derived from using any state-of-the-art approach as inputs, and a set of attributed test cases. As an output a list of allocated test cases for each product is obtained. We also define a novel fitness function to guide the search and we propose corresponding crossover and mutation operators. The search and test process is iteratively repeated until the time budget is consumed. We performed an evaluation with a CPSPL as a case study. Results suggest that our approach can reduce the fault detection time by 61% and 65% on average when compared with the traditional test process and the Random Search algorithm respectively.

[1]  Aitor Arrieta,et al.  Search-based test case selection of cyber-physical system product lines for simulation-based validation , 2016, SPLC.

[2]  Sergio Segura,et al.  SIP: Optimal Product Selection from Feature Models Using Many-Objective Evolutionary Optimization , 2016, ACM Trans. Softw. Eng. Methodol..

[3]  Lionel C. Briand,et al.  Automated Test Suite Generation for Time-Continuous Simulink Models , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[4]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[5]  Jacques Klein,et al.  Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines , 2014, IEEE Transactions on Software Engineering.

[6]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[7]  Sergio Segura,et al.  Automated analysis of feature models 20 years later: A literature review , 2010, Inf. Syst..

[8]  John D. McGregor,et al.  On strategies for testing software product lines: A systematic literature review , 2014, Inf. Softw. Technol..

[9]  Pieter J. Mosterman,et al.  Automating humanitarian missions with a heterogeneous fleet of vehicles , 2014, Annu. Rev. Control..

[10]  Antonio Ruiz Cortés,et al.  Multi-objective test case prioritization in highly configurable systems: A case study , 2016, J. Syst. Softw..

[11]  Aitor Arrieta,et al.  Automatic generation of test system instances for configurable cyber-physical systems , 2016, Software Quality Journal.

[12]  Gordon Fraser,et al.  On Parameter Tuning in Search Based Software Engineering , 2011, SSBSE.

[13]  Arnaud Gotlieb,et al.  A systematic test case selection methodology for product lines: results and insights from an industrial case study , 2014, Empirical Software Engineering.

[14]  Sebastian Oster,et al.  Automated Incremental Pairwise Testing of Software Product Lines , 2010, SPLC.

[15]  Gunter Saake,et al.  Similarity-based prioritization in software product-line testing , 2014, SPLC.

[16]  Arnaud Gotlieb,et al.  Practical pairwise testing for software product lines , 2013, SPLC '13.

[17]  Arnaud Gotlieb,et al.  Multi-objective test prioritization in software product line testing: an industrial case study , 2014, SPLC.

[18]  Malte Lochau,et al.  MoSo-PoLiTe: tool support for pairwise and model-based software product line testing , 2011, VaMoS '11.

[19]  Yuanyuan Zhang,et al.  Search based software engineering for software product line engineering: a survey and directions for future work , 2014, SPLC.

[20]  Sergio Segura,et al.  Variability testing in the wild: the Drupal case study , 2017, Software & Systems Modeling.

[21]  Arnaud Gotlieb,et al.  Automated product line test case selection: industrial case study and controlled experiment , 2017, Software & Systems Modeling.

[22]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[23]  Alexander Egyed,et al.  A first systematic mapping study on combinatorial interaction testing for software product lines , 2015, 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[24]  Sandro Schulze,et al.  Risk-based integration testing of software product lines , 2017, VaMoS.

[25]  Aitor Arrieta,et al.  Search-based test case generation for Cyber-Physical Systems , 2017, 2017 IEEE Congress on Evolutionary Computation (CEC).

[26]  Arnaud Gotlieb,et al.  Cost-effective test suite minimization in product lines using search techniques , 2015, J. Syst. Softw..

[27]  Øystein Haugen,et al.  An algorithm for generating t-wise covering arrays from large feature models , 2012, SPLC '12.

[28]  Alexander Egyed,et al.  A Hitchhiker's Guide to Search-Based Software Engineering for Software Product Lines , 2014, ArXiv.

[29]  Arnaud Gotlieb,et al.  Minimizing test suites in software product lines using weight-based genetic algorithms , 2013, GECCO '13.

[30]  Klaus Pohl,et al.  Software Product Line Engineering , 2005 .

[31]  Gunter Saake,et al.  A Classification and Survey of Analysis Strategies for Software Product Lines , 2014, ACM Comput. Surv..

[32]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[33]  Alexander Egyed,et al.  A systematic mapping study of search-based software engineering for software product lines , 2015, Inf. Softw. Technol..

[34]  Yves Le Traon,et al.  Threats to the validity of mutation-based test assessment , 2016, ISSTA.

[35]  Aitor Arrieta,et al.  Test Case Prioritization of Configurable Cyber-Physical Systems with Weight-Based Search Algorithms , 2016, GECCO.

[36]  Jacques Klein,et al.  Pairwise testing for software product lines: comparison of two approaches , 2012, Software Quality Journal.

[37]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[38]  Gunter Saake,et al.  Effective product-line testing using similarity-based product prioritization , 2016, Software & Systems Modeling.

[39]  John D. McGregor,et al.  A systematic mapping study of software product lines testing , 2011, Inf. Softw. Technol..

[40]  Mustafa Al-Hajjaji,et al.  Optimizing product orders using graph algorithms for improving incremental product-line analysis , 2017, VaMoS.

[41]  Klaus Pohl,et al.  Avoiding Redundant Testing in Application Engineering , 2010, SPLC.

[42]  Alexander Egyed,et al.  Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges , 2016, Computational Intelligence and Quantitative Software Engineering.

[43]  Sergio Segura,et al.  A Comparison of Test Case Prioritization Criteria for Software Product Lines , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.