Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges

Because of economical, technological and marketing reasons today’s software systems are more frequently being built as families where each product variant implements a different combination of features. Software families are commonly called Software Product Lines (SPLs) and over the past three decades have been the subject of extensive research and application. Among the benefits of SPLs are: increased software reuse, faster and easier product customization, and reduced time to market. However, testing SPLs is specially challenging as the number of product variants is usually large making it infeasible to test every single variant. In recent years there has been an increasing interest in applying evolutionary computation techniques for SPL testing. In this chapter, we provide a concise overview of the state of the art and practice in SPL testing with evolutionary techniques as well as to highlight open questions and areas for future research.

[1]  Antonio J. Nebro,et al.  jMetal: A Java framework for multi-objective optimization , 2011, Adv. Eng. Softw..

[2]  Øystein Haugen,et al.  An algorithm for generating t-wise covering arrays from large feature models , 2012, SPLC '12.

[3]  Enrique Alba,et al.  On the Application of SAT Solvers to the Test Suite Minimization Problem , 2012, SSBSE.

[4]  Jacques Klein,et al.  Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines , 2014, IEEE Transactions on Software Engineering.

[5]  Tim Menzies,et al.  Scalable product line configuration: A straw to break the camel's back , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[6]  Sebastian Oster,et al.  Automated Incremental Pairwise Testing of Software Product Lines , 2010, SPLC.

[7]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[8]  Gunter Saake,et al.  Similarity-based prioritization in software product-line testing , 2014, SPLC.

[9]  Jacques Klein,et al.  Multi-objective test generation for software product lines , 2013, SPLC '13.

[10]  Alexander Egyed,et al.  Applying multiobjective evolutionary algorithms to dynamic software product lines for reconfiguring mobile applications , 2015, J. Syst. Softw..

[11]  Alexandr Murashkin,et al.  Visualization and exploration of optimal variants in product line engineering , 2013, SPLC '13.

[12]  Roberto Erick Lopez-Herrejon,et al.  A Standard Problem for Evaluating Product-Line Methodologies , 2001, GCSE.

[13]  Jasbir S. Arora,et al.  Survey of multi-objective optimization methods for engineering , 2004 .

[14]  Enrique Alba,et al.  A parallel evolutionary algorithm for prioritized pairwise testing of software product lines , 2014, GECCO.

[15]  Tim Menzies,et al.  On the value of user preferences in search-based software engineering: A case study in software product lines , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[16]  Myra B. Cohen,et al.  Moving Forward with Combinatorial Interaction Testing , 2014, Computer.

[17]  Mario Cannataro,et al.  Protein-to-protein interactions: Technologies, databases, and algorithms , 2010, CSUR.

[18]  John D. McGregor,et al.  Strategies for testing products in software product lines , 2012, ACM SIGSOFT Softw. Eng. Notes.

[19]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[20]  Arnaud Gotlieb,et al.  Multi-objective test prioritization in software product line testing: an industrial case study , 2014, SPLC.

[21]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[22]  Alexander Egyed,et al.  Enhancing Clone-and-Own with Systematic Reuse for Developing Software Variants , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[23]  Andy Schürr,et al.  Model-based pairwise testing for feature interaction coverage in software product line engineering , 2011, Software Quality Journal.

[24]  Jacques Klein,et al.  Pairwise testing for software product lines: comparison of two approaches , 2012, Software Quality Journal.

[25]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[26]  Sergio Segura,et al.  Automated analysis of feature models 20 years later: A literature review , 2010, Inf. Syst..

[27]  Enrique Alba,et al.  Evolutionary algorithm for prioritized pairwise test data generation , 2012, GECCO '12.

[28]  Krzysztof Czarnecki,et al.  Efficient synthesis of feature models , 2014, Inf. Softw. Technol..

[29]  Per Runeson,et al.  Testing Software Product Lines , 2011, IEEE Software.

[30]  Alexander Egyed,et al.  Towards interactive visualization support for pairwise testing software product lines , 2013, 2013 First IEEE Working Conference on Software Visualization (VISSOFT).

[31]  G. Nemhauser,et al.  Integer Programming , 2020 .

[32]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[33]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[34]  John D. McGregor,et al.  On strategies for testing software product lines: A systematic literature review , 2014, Inf. Softw. Technol..

[35]  Lianping Chen,et al.  A systematic review of evaluation of variability management approaches in software product lines , 2011, Inf. Softw. Technol..

[36]  Sungwon Kang,et al.  A survey on software product line testing , 2012, SPLC '12.

[37]  John D. McGregor,et al.  Corrigendum to: "A systematic mapping study of software product lines testing" [Inf. Softw. Technology 53 (5) (2011) 407-423] , 2012, Information and Software Technology.

[38]  Derek Rayside,et al.  Comparison of exact and approximate multi-objective optimization for software product lines , 2014, SPLC.

[39]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[40]  Arnaud Gotlieb,et al.  Minimizing test suites in software product lines using weight-based genetic algorithms , 2013, GECCO '13.

[41]  Angelo Gargantini,et al.  Combinatorial Testing for Feature Models Using CitLab , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[42]  Alexander Egyed,et al.  Using feature model knowledge to speed up the generation of covering arrays , 2013, VaMoS.

[43]  Alexander Egyed,et al.  A Hitchhiker's Guide to Search-Based Software Engineering for Software Product Lines , 2014, ArXiv.

[44]  Yves Le Traon,et al.  Mutation-Based Generation of Software Product Line Test Configurations , 2014, SSBSE.

[45]  Eckart Zitzler,et al.  Evolutionary multiobjective optimization , 2007, GECCO '07.

[46]  Sergio Segura,et al.  An assessment of search-based techniques for reverse engineering feature models , 2015, J. Syst. Softw..

[47]  Arnaud Gotlieb,et al.  PACOGEN: Automatic Generation of Pairwise Test Configurations from Feature Models , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[48]  Alexander Egyed,et al.  A systematic mapping study of search-based software engineering for software product lines , 2015, Inf. Softw. Technol..

[49]  Alexander Egyed,et al.  Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of Software Product Lines , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[50]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[51]  Claes Wohlin,et al.  On the reliability of mapping studies in software engineering , 2013, J. Syst. Softw..

[52]  Gunter Saake,et al.  A Classification and Survey of Analysis Strategies for Software Product Lines , 2014, ACM Comput. Surv..

[53]  Jerffeson Teixeira de Souza,et al.  Ten Years of Search Based Software Engineering: A Bibliometric Analysis , 2011, SSBSE.

[54]  Per Runeson,et al.  Software product line testing - A systematic mapping study , 2011, Inf. Softw. Technol..

[55]  Klaus Pohl,et al.  Software Product Line Engineering , 2005 .

[56]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems (Genetic and Evolutionary Computation) , 2006 .

[57]  Kalyanmoy Deb,et al.  Multi-objective optimization using evolutionary algorithms , 2001, Wiley-Interscience series in systems and optimization.

[58]  Kyo Chul Kang,et al.  Feature-Oriented Domain Analysis (FODA) Feasibility Study , 1990 .

[59]  Alexander Egyed,et al.  Multi-objective Optimal Test Suite Computation for Software Product Line Pairwise Testing , 2013, 2013 IEEE International Conference on Software Maintenance.

[60]  Don Batory,et al.  Scaling Step-Wise Refinement , 2004, IEEE Trans. Software Eng..

[61]  Alexander Egyed,et al.  Feature Model Synthesis with Genetic Programming , 2014, SSBSE.

[62]  Sergio Segura,et al.  A Comparison of Test Case Prioritization Criteria for Software Product Lines , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[63]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[64]  Alexander Egyed,et al.  Towards a Benchmark and a Comparison Framework for Combinatorial Interaction Testing of Software Product Lines , 2014, ArXiv.

[65]  Dragan Gasevic,et al.  Evolutionary Search-Based Test Generation for Software Product Line Feature Models , 2012, CAiSE.

[66]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[67]  Ricardo A. L. Rabêlo,et al.  Toward a hybrid approach to generate Software Product Line portfolios , 2013, 2013 IEEE Congress on Evolutionary Computation.

[68]  Andrew M. Sutton,et al.  A polynomial time computation of the exact correlation structure of k-satisfiability landscapes , 2009, GECCO '09.

[69]  Krzysztof Czarnecki,et al.  An Exploratory Study of Cloning in Industrial Software Product Lines , 2013, 2013 17th European Conference on Software Maintenance and Reengineering.

[70]  Antonio Ruiz Cortés,et al.  FAMA Framework , 2008, 2008 12th International Software Product Line Conference.

[71]  Alexander Egyed,et al.  Improving CASA Runtime Performance by Exploiting Basic Feature Model Analysis , 2013, ArXiv.

[72]  Andy Schürr,et al.  Model-based coverage-driven test suite generation for software product lines , 2011, MODELS'11.

[73]  Jacques Klein,et al.  PLEDGE: a product line editor and test generation tool , 2013, SPLC '13 Workshops.

[74]  Myra B. Cohen,et al.  Continuous test suite augmentation in software product lines , 2013, SPLC '13.