A tuned version of genetic algorithm for efficient test suite generation in interactive t-way testing strategy

Abstract Context To improve the quality and correctness of a software product it is necessary to test different aspects of the software system. Among different approaches for software testing, combinatorial testing along with covering array is a proper testing method. The most challenging problem in combinatorial testing strategies like t-way, is the combinatorial explosion which considers all combinations of input parameters. Many evolutionary and meta-heuristic strategies have been proposed to address and mitigate this problem. Objective Genetic Algorithm (GA) is an evolutionary search-based technique that has been used in t-way interaction testing by different approaches. Although useful, all of these approaches can produce test suite with small interaction strengths (i.e. t ≤ 6). Additionally, most of them suffer from expensive computations. Even though there are other strategies which use different meta-heuristic algorithms to solve these problems, in this paper, we propose an efficient uniform and variable t-way minimal test suite generation approach to address these problems using GA, called Genetic Strategy (GS). Method By changing the bit structure and accessing test cases quickly, GS improves performance of the fitness function. These adjustments and reduction of the complexities of GA in the proposed GS decreases the test suite size and increases the speed of test suite generation up to t = 20 . Results To evaluate the efficiency and performance of the proposed GS, various experiments are performed on different set of benchmarks. Experimental results show that not only GS supports higher interaction strengths in comparison with the existing GA-based strategies, but also its supported interaction strength is higher than most of other AI-based and computational-based strategies. Conclusion Furthermore, experimental results show that GS can compete against the existing (both AI-based and computational-based) strategies in terms of efficiency and performance in most of the case studies.

[1]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[2]  D. Richard Kuhn,et al.  Tower of covering arrays , 2015 .

[3]  Chee Peng Lim,et al.  Application of Particle Swarm Optimization to uniform and variable strength covering array construction , 2012, Appl. Soft Comput..

[4]  Yu Lei,et al.  Refining the In-Parameter-Order Strategy for Constructing Covering Arrays , 2008, Journal of research of the National Institute of Standards and Technology.

[5]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[6]  Soumen Maity,et al.  An Improved Test Generation Algorithm for PairWise Testing , 2003 .

[7]  Kamal Zuhairi Zamli,et al.  A variable strength interaction test suites generation strategy using Particle Swarm Optimization , 2011, J. Syst. Softw..

[8]  A. Hartman.  Software and Hardware Testing Using Combinatorial Covering Suites , 2005 .

[9]  Charles J. Colbourn,et al.  A density-based greedy algorithm for higher strength covering arrays , 2009 .

[10]  J. Czerwonka.  Pairwise Testing in Real World Practical Extensions to Test Case Generators , 2006 .

[11]  Bestoun S. Ahmed,et al.  Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm , 2015, Inf. Softw. Technol..

[12]  Angelo Gargantini,et al.  IPO-s: Incremental Generation of Combinatorial Interaction Test Data Based on Symmetries of Covering Arrays , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[13]  Mohsen Rahmani,et al.  A heuristic solution for model checking graph transformation systems , 2014, Appl. Soft Comput..

[14]  Mohd Hafiz Mohd Hassin,et al.  tReductSA - Test Redundancy Reduction Strategy Based on Simulated Annealing , 2014, SoMeT.

[15]  Rusli Abdullah,et al.  Design and implementation of a t-way test data generation strategy with automated execution tool support , 2011, Inf. Sci..

[16]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[17]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[18]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[19]  Z. ZamliKamal,et al.  A Cuckoo Search Based Pairwise Strategy ForCombinatorial Testing Problem , 2015 .

[20]  Baowen Xu,et al.  Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite , 2008, 2008 The Eighth International Conference on Quality Software.

[21]  David E. Goldberg,et al.  Genetic Algorithms, Tournament Selection, and the Effects of Noise , 1995, Complex Syst..

[22]  Harmen-Hinrich Sthamer,et al.  The automatic generation of software test data using genetic algorithms , 1995 .

[23]  Kamal Z. Zamli,et al.  Interaction test data generation using Harmony Search Algorithm , 2011, 2011 IEEE Symposium on Industrial Electronics and Applications.

[24]  Xiang Chen,et al.  Variable Strength Interaction Testing with an Ant Colony System Approach , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[25]  Sangeeta Sabharwal,et al.  An Approach to Test Set Generation for Pair-Wise Testing Using Genetic Algorithms , 2013, SSBSE.

[26]  Bestoun S. Ahmed,et al.  An efficient strategy for covering array construction with fuzzy logic-based adaptive swarm optimization for software testing use , 2015, Expert Syst. Appl..

[27]  Charles J. Colbourn,et al.  One-test-at-a-time heuristic search for interaction test suites , 2007, GECCO '07.

[28]  Yu Lei,et al.  IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing , 2008 .

[29]  Wolfgang Banzhaf,et al.  Genetic Programming: An Introduction , 1997 .

[30]  Z. ZamliKamal,et al.  Tuning of Cuckoo Search Based Strategy for T-Way Testing , 2015 .

[31]  Yoonsik Cheon,et al.  PWiseGen: Generating test cases for pairwise testing using genetic algorithms , 2011, 2011 IEEE International Conference on Computer Science and Automation Engineering.

[32]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[33]  Jun Yan,et al.  Generating combinatorial test suite using combinatorial optimization , 2014, J. Syst. Softw..

[34]  Graham Kendall,et al.  A Tabu Search hyper-heuristic strategy for t-way test suite generation , 2016, Appl. Soft Comput..

[35]  Soumen Maity,et al.  Improved test generation algorithms for pair-wise testing , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[36]  Charles J. Colbourn,et al.  The density algorithm for pairwise interaction testing: Research Articles , 2007 .

[37]  Vahid Rafe,et al.  An Optimal Solution for Test Case Generation Using ROBDD Graph and PSO Algorithm , 2016, Qual. Reliab. Eng. Int..

[38]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[39]  Vahid Rafe.  Scenario-driven analysis of systems specified through graph transformations , 2013, J. Vis. Lang. Comput..

[40]  Fadhl Hujainah,et al.  Sequence and sequence-less T-way test suite generation strategy based on flower pollination algorithm , 2015, 2015 IEEE Student Conference on Research and Development (SCOReD).

[41]  Myra B. Cohen,et al.  An Improved Meta-heuristic Search for Constrained Interaction Testing , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[42]  James D. McCaffrey,et al.  An Empirical Study of Pairwise Test Set Generation Using a Genetic Algorithm , 2010, 2010 Seventh International Conference on Information Technology: New Generations.

[43]  Changhai Nie,et al.  A Discrete Particle Swarm Optimization for Covering Array Generation , 2015, IEEE Transactions on Evolutionary Computation.

[44]  Charles J. Colbourn,et al.  The density algorithm for pairwise interaction testing , 2007, Softw. Test. Verification Reliab..

[45]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[46]  Kamal Z. Zamli,et al.  Design and implementation of a harmony-search-based variable-strength t-way testing strategy with constraints support , 2012, Inf. Softw. Technol..

[47]  Sangeeta Sabharwal,et al.  Construction of Mixed Covering Arrays for Pair-wise Testing Using Probabilistic Approach in Genetic Algorithm , 2016 .

[48]  Sangeeta Sabharwal,et al.  Construction of Variable Strength Covering Array for Combinatorial Testing Using a Greedy Approach to Genetic Algorithm , 2015, e Informatica Softw. Eng. J..

[49]  Z. ZamliKamal,et al.  On Test Case Generation Satisfying the MC/DC Criterion , 2013 .

[50]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[51]  Yu-Wen Tung,et al.  Automating test case generation for the new generation mission software system , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[52]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[53]  Myra B. Cohen,et al.  Designing Test Suites for Software Interactions Testing , 2004 .

[54]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[55]  Kamal Z. Zamli,et al.  SPLBA: An interaction strategy for testing software product lines using the Bat-inspired algorithm , 2015, 2015 4th International Conference on Software Engineering and Computer Systems (ICSECS).

[56]  Myra B. Cohen,et al.  Constructing strength three covering arrays with augmented annealing , 2003, Discret. Math..

[57]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[58]  Brett Stevens,et al.  Efficient software testing protocols , 1998, CASCON.

[59]  Ysong Yueh Yu,et al.  Generating, selecting and prioritizing test cases from specifications with tool support , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[60]  J. Wegener,et al.  Test Case Design by Means of the CTE XL , 2000 .

[61]  Kaile Su,et al.  TCA: An Efficient Two-Mode Meta-Heuristic Algorithm for Combinatorial Test Generation (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[62]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[63]  Kamal Zuhairi Zamli,et al.  Assessing optimization based strategies for t-way test suite generation: The case for flower-based strategy , 2015, 2015 IEEE International Conference on Control System, Computing and Engineering (ICCSCE).