A Tabu Search hyper-heuristic strategy for t-way test suite generation

Display Omitted HHH is the first strategy that adopts the hyper-heuristic approach for t-way test suite generationHHH introduces new approach for the heuristic selection and move acceptance mechanism based on three operators (i.e. improvement operator, diversify operator, and intensify operator) that are integrated into the Tabu search HLH.HHH outperforms existing strategies as far as optimality of test suite is concerned in many benchmarks. This paper proposes a novel hybrid t-way test generation strategy (where t indicates interaction strength), called High Level Hyper-Heuristic (HHH). HHH adopts Tabu Search as its high level meta-heuristic and leverages on the strength of four low level meta-heuristics, comprising of Teaching Learning based Optimization, Global Neighborhood Algorithm, Particle Swarm Optimization, and Cuckoo Search Algorithm. HHH is able to capitalize on the strengths and limit the deficiencies of each individual algorithm in a collective and synergistic manner. Unlike existing hyper-heuristics, HHH relies on three defined operators, based on improvement, intensification and diversification, to adaptively select the most suitable meta-heuristic at any particular time. Our results are promising as HHH manages to outperform existing t-way strategies on many of the benchmarks.

[1]  Graham Kendall,et al.  A Classification of Hyper-heuristic Approaches , 2010 .

[2]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[3]  Matej Crepinsek,et al.  A note on teaching-learning-based optimization algorithm , 2012, Inf. Sci..

[4]  Bo Xing,et al.  Innovative Computational Intelligence: A Rough Guide to 134 Clever Algorithms , 2013 .

[5]  Xin-She Yang,et al.  Cuckoo Search via Lévy flights , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[6]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[7]  Graham Kendall,et al.  Hyper-Heuristics: An Emerging Direction in Modern Search Technology , 2003, Handbook of Metaheuristics.

[8]  Kamal Z. Zamli,et al.  MIPOG - An Efficient t-Way Minimization Strategy for Combinatorial Testing , 2011 .

[9]  Charles J. Colbourn,et al.  Strength two covering arrays: Existence tables and projection , 2008, Discret. Math..

[10]  J. Czerwonka Pairwise Testing in the Real World : Practical Extensions to Test-Case Scenarios , 2011 .

[11]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[12]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[13]  Kamal Z. Zamli,et al.  The Development of a Particle Swarm Based Optimization Strategy for Pairwise Testing , 2011 .

[14]  Marjan Mernik,et al.  Is a comparison of results meaningful from the inexact replications of computational experiments? , 2016, Soft Comput..

[15]  Kamal Z. Zamli,et al.  Design and implementation of a harmony-search-based variable-strength t-way testing strategy with constraints support , 2012, Inf. Softw. Technol..

[16]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[17]  J. Wegener,et al.  Test Case Design by Means of the CTE XL , 2000 .

[18]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[19]  K. A. Bush Orthogonal Arrays of Index Unity , 1952 .

[20]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[21]  Charles J. Colbourn,et al.  One-test-at-a-time heuristic search for interaction test suites , 2007, GECCO '07.

[22]  Yu Lei,et al.  IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing , 2008 .

[23]  Graham Kendall,et al.  Automatic Design of a Hyper-Heuristic Framework With Gene Expression Programming for Combinatorial Optimization Problems , 2015, IEEE Transactions on Evolutionary Computation.

[24]  Graham Kendall,et al.  A Dynamic Multiarmed Bandit-Gene Expression Programming Hyper-Heuristic for Combinatorial Optimization Problems , 2015, IEEE Transactions on Cybernetics.

[25]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[26]  Brett Stevens,et al.  Efficient software testing protocols , 1998, CASCON.

[27]  Graham Kendall,et al.  A Hyperheuristic Approach to Scheduling a Sales Summit , 2000, PATAT.

[28]  Rusli Abdullah,et al.  Design and implementation of a t-way test data generation strategy with automated execution tool support , 2011, Inf. Sci..

[29]  R. Venkata Rao,et al.  Teaching-learning-based optimization: A novel method for constrained mechanical design optimization problems , 2011, Comput. Aided Des..

[30]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[31]  Myra B. Cohen,et al.  Constructing strength three covering arrays with augmented annealing , 2003, Discret. Math..

[32]  Michel Gendreau,et al.  Hyper-heuristics: a survey of the state of the art , 2013, J. Oper. Res. Soc..

[33]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[34]  Donald L. Kreher,et al.  On the state of strength‐three covering arrays , 2002 .

[35]  Yuanyuan Zhang,et al.  Search based software engineering for software product line engineering: a survey and directions for future work , 2014, SPLC.

[36]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[37]  El-Ghazali Talbi,et al.  A Unified Taxonomy of Hybrid Metaheuristics with Mathematical Programming, Constraint Programming and Machine Learning , 2013, Hybrid Metaheuristics.

[38]  Amer Draa,et al.  On the performances of the flower pollination algorithm - Qualitative and quantitative analyses , 2015, Appl. Soft Comput..

[39]  Graham Kendall,et al.  A Tabu-Search Hyperheuristic for Timetabling and Rostering , 2003, J. Heuristics.

[40]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[41]  Marjan Mernik,et al.  Exploration and exploitation in evolutionary algorithms: A survey , 2013, CSUR.

[42]  Kamal Z. Zamli,et al.  MC‐MIPOG: A Parallel t‐Way Test Generation Strategy for Multicore Systems , 2010 .

[43]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[44]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[45]  Jeff Yu Lei,et al.  IPOG/IPOG‐D: efficient test generation for multi‐way combinatorial testing , 2008, Softw. Test. Verification Reliab..

[46]  Chee Peng Lim,et al.  Application of Particle Swarm Optimization to uniform and variable strength covering array construction , 2012, Appl. Soft Comput..

[47]  Yu Lei,et al.  Refining the In-Parameter-Order Strategy for Constructing Covering Arrays , 2008, Journal of research of the National Institute of Standards and Technology.

[48]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[49]  Kamal Z. Zamli,et al.  T-Way Test Data Generation Strategy Based on Particle Swarm Optimization , 2010, 2010 Second International Conference on Computer Research and Development.

[50]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[51]  Kamal Z. Zamli,et al.  PSTG: A T-Way Strategy Adopting Particle Swarm Optimization , 2010, 2010 Fourth Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation.

[52]  Ysong Yueh Yu,et al.  Generating, selecting and prioritizing test cases from specifications with tool support , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[53]  Dervis Karaboga,et al.  On clarifying misconceptions when comparing variants of the Artificial Bee Colony Algorithm by offering a new implementation , 2015, Inf. Sci..

[54]  Xin-She Yang,et al.  Nature-Inspired Metaheuristic Algorithms: Second Edition , 2010 .

[55]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[56]  Marjan Mernik,et al.  Replication and comparison of computational experiments in applied evolutionary computing: Common pitfalls and guidelines to avoid them , 2014, Appl. Soft Comput..

[57]  Graham Kendall,et al.  A Monte Carlo Hyper-Heuristic To Optimise Component Placement Sequencing For Multi Head Placement Machine , 2003 .

[58]  Chee Peng Lim,et al.  Constructing a t-way interaction test suite using the Particle Swarm Optimization approach , 2012 .

[59]  Myra B. Cohen,et al.  Designing Test Suites for Software Interactions Testing , 2004 .

[60]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[61]  Bestoun S. Ahmed,et al.  Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm , 2015, Inf. Softw. Technol..

[62]  Rusli Abdullah,et al.  G2Way A Backtracking Strategy for Pairwise Test Data Generation , 2008, 2008 15th Asia-Pacific Software Engineering Conference.

[63]  Kamal Zuhairi Zamli,et al.  A variable strength interaction test suites generation strategy using Particle Swarm Optimization , 2011, J. Syst. Softw..

[64]  Harold W. Lewis,et al.  A New Optimization Algorithm For Combinatorial Problems , 2013 .