Search Based Combinatorial Testing

Search techniques can dramatically change our ability to solve a host of problems in applied science and engineering, many search techniques have been developed and applied successfully in many fields, including search based software engineering (SBSE). As a key problem of combinatorial testing, covering array generation has been widely studied and many search techniques have been applied which can be named as search based combinatorial testing (SBCT). SBCT is a branch of search based software testing (SBST) within SBSE. In this paper, to explore the applicability and effectiveness of SBCT, we design six variants from existing search algorithms: Genetic Algorithm, Particle Swarm Optimization and Ant Colony Algorithm by reversing and randomizing their mechanisms. We study their effectiveness in terms of generating a covering array and compare their performance. Experiments show that these search techniques can work well with distinct performance in covering array generation. We believe that these search techniques can be further improved by fine-tuning their configuration and used in broad ranges of area.

[1]  Charles J. Colbourn,et al.  Tabu search for covering arrays using permutation vectors , 2009 .

[2]  James D. McCaffrey,et al.  An Empirical Study of Pairwise Test Set Generation Using a Genetic Algorithm , 2010, 2010 Seventh International Conference on Information Technology: New Generations.

[3]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[4]  Myra B. Cohen,et al.  Constructing strength three covering arrays with augmented annealing , 2003, Discret. Math..

[5]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[6]  Kamal Z. Zamli,et al.  The Development of a Particle Swarm Based Optimization Strategy for Pairwise Testing , 2011 .

[7]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[8]  Hareton K. N. Leung,et al.  Empirically Identifying the Best Greedy Algorithm for Covering Array Generation , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[9]  Moataz A. Ahmed,et al.  Pair-wise test coverage using genetic algorithms , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[10]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[11]  Xiang Chen,et al.  Building Prioritized Pairwise Interaction Test Suites with Ant Colony Optimization , 2009, 2009 Ninth International Conference on Quality Software.

[12]  Xiang Chen,et al.  Variable Strength Interaction Testing with an Ant Colony System Approach , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[13]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[14]  José Torres-Jiménez,et al.  New bounds for binary covering arrays using simulated annealing , 2012, Inf. Sci..

[15]  José Torres-Jiménez,et al.  Simulated Annealing for constructing binary covering arrays of variable strength , 2010, IEEE Congress on Evolutionary Computation.

[16]  Chee Peng Lim,et al.  Application of Particle Swarm Optimization to uniform and variable strength covering array construction , 2012, Appl. Soft Comput..

[17]  Gregory M. Kapfhammer,et al.  Empirically Identifying the Best Genetic Algorithm for Covering Array Generation , 2011 .

[18]  Xiang Chen,et al.  Applying Particle Swarm Optimization to Pairwise Testing , 2010, 2010 IEEE 34th Annual Computer Software and Applications Conference.

[19]  Kamal Z. Zamli,et al.  PSTG: A T-Way Strategy Adopting Particle Swarm Optimization , 2010, 2010 Fourth Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation.

[20]  Kamal Zuhairi Zamli,et al.  A variable strength interaction test suites generation strategy using Particle Swarm Optimization , 2011, J. Syst. Softw..

[21]  C. Colbourn.  Combinatorial aspects of covering arrays , 2006 .

[22]  José Torres-Jiménez,et al.  Memetic Algorithms for Constructing Binary Covering Arrays of Strength Three , 2009, Artificial Evolution.

[23]  Phil McMinn,et al.  Search-Based Software Testing: Past, Present and Future , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[24]  Charles J. Colbourn,et al.  One-test-at-a-time heuristic search for interaction test suites , 2007, GECCO '07.

[25]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[26]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..