Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm

A new approach is used to generate combinatorial test suites.Cuckoo Search application is investigated for a new type of application and case study.The strategy opens a new approach in Search Based Software Testing (SBST).The strategy is evaluated through different benchmarks and it is able to get comparative results.Application of combinatorial optimization is also investigated in the current paper. ContextSoftware has become an innovative solution nowadays for many applications and methods in science and engineering. Ensuring the quality and correctness of software is challenging because each program has different configurations and input domains. To ensure the quality of software, all possible configurations and input combinations need to be evaluated against their expected outputs. However, this exhaustive test is impractical because of time and resource constraints due to the large domain of input and configurations. Thus, different sampling techniques have been used to sample these input domains and configurations. ObjectiveCombinatorial testing can be used to effectively detect faults in software-under-test. This technique uses combinatorial optimization concepts to systematically minimize the number of test cases by considering the combinations of inputs. This paper proposes a new strategy to generate combinatorial test suite by using Cuckoo Search concepts. MethodCuckoo Search is used in the design and implementation of a strategy to construct optimized combinatorial sets. The strategy consists of different algorithms for construction. These algorithms are combined to serve the Cuckoo Search. ResultsThe efficiency and performance of the new technique were proven through different experiment sets. The effectiveness of the strategy is assessed by applying the generated test suites on a real-world case study for the purpose of functional testing. ConclusionResults show that the generated test suites can detect faults effectively. In addition, the strategy also opens a new direction for the application of Cuckoo Search in the context of software engineering.

[1]  Chee Peng Lim,et al.  Application of Particle Swarm Optimization to uniform and variable strength covering array construction , 2012, Appl. Soft Comput..

[2]  Zahra Moravej,et al.  A novel approach based on cuckoo search for DG allocation in distribution network , 2013 .

[3]  Yu Lei,et al.  Refining the In-Parameter-Order Strategy for Constructing Covering Arrays , 2008, Journal of research of the National Institute of Standards and Technology.

[4]  Xin-She Yang,et al.  Nature-Inspired Metaheuristic Algorithms , 2008 .

[5]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[6]  Mario Cannataro,et al.  Protein-to-protein interactions: Technologies, databases, and algorithms , 2010, CSUR.

[7]  Chee Peng Lim,et al.  Constructing a t-way interaction test suite using the Particle Swarm Optimization approach , 2012 .

[8]  Jun Yan,et al.  Generating combinatorial test suite using combinatorial optimization , 2014, J. Syst. Softw..

[9]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[10]  Charles J. Colbourn,et al.  Covering and radius-covering arrays: Constructions and classification , 2010, Discret. Appl. Math..

[11]  Noritaka Kobayashi,et al.  Design and Evaluation of Automatic Test Generation Strategies for Functional Testing of Software , 2002 .

[12]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[13]  Charles J. Colbourn,et al.  Experimental designs in software engineering: d-optimal designs and covering arrays , 2004, WISER '04.

[14]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[15]  Baowen Xu,et al.  Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite , 2008, 2008 The Eighth International Conference on Quality Software.

[16]  Xin-She Yang,et al.  Discrete cuckoo search algorithm for the travelling salesman problem , 2014, Neural Computing and Applications.

[17]  Angelo Gargantini,et al.  IPO-s: Incremental Generation of Combinatorial Interaction Test Data Based on Symmetries of Covering Arrays , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[18]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[19]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[20]  Asha Gowda Karegowda,et al.  A Brief Survey on Cuckoo Search Applications , 2014 .

[21]  Patrick J. Schroeder,et al.  Comparing the fault detection effectiveness of n-way and random test suites , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[22]  Seyed Javad Mirabedini,et al.  Combining Cuckoo and Tabu Algorithms for Solving Quadratic Assignment Problems , 2012 .

[23]  Iztok Fister,et al.  A comprehensive review of cuckoo search: variants and hybrids , 2013, Int. J. Math. Model. Numer. Optimisation.

[24]  Xin-She Yang,et al.  Engineering optimisation by cuckoo search , 2010 .

[25]  Soumen Maity,et al.  An Improved Test Generation Algorithm for PairWise Testing , 2003 .

[26]  Bestoun S. Ahmed,et al.  Generating combinatorial test cases using Simplified Swarm Optimization (SSO) algorithm for automated GUI functional testing , 2014 .

[27]  Yu Lei,et al.  IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing , 2008 .

[28]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[29]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[30]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[31]  Xiao Qu,et al.  Testing of Configurable Systems , 2013, Adv. Comput..

[32]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[33]  Robert L. Probert,et al.  A practical strategy for testing pair-wise coverage of network interfaces , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[34]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[35]  Ali Kaveh,et al.  Optimum design of steel frames using Cuckoo Search algorithm with Lévy flights , 2013 .

[36]  Victor V. Kuliamin,et al.  A survey of methods for constructing covering arrays , 2011, Programming and Computer Software.

[37]  Myra B. Cohen,et al.  Designing Test Suites for Software Interactions Testing , 2004 .

[38]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[39]  Thang Trung Nguyen,et al.  Cuckoo search algorithm for short-term hydrothermal scheduling , 2014 .

[40]  Seif-Eddeen K. Fateen,et al.  Cuckoo Search: A new nature-inspired optimization method for phase equilibrium calculations , 2013 .

[41]  Douglas C. Montgomery,et al.  Software performance testing using covering arrays: efficient screening designs with categorical factors , 2005, WOSP '05.

[42]  Anil Kumar,et al.  Design optimization for reliable embedded system using Cuckoo Search , 2011, 2011 3rd International Conference on Electronics Computer Technology.

[43]  Charles J. Colbourn,et al.  The density algorithm for pairwise interaction testing , 2007, Softw. Test. Verification Reliab..

[44]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[45]  Charles J. Colbourn,et al.  The density algorithm for pairwise interaction testing: Research Articles , 2007 .

[46]  Charles J. Colbourn,et al.  Merging covering arrays and compressing multiple sequence alignments , 2009, Discret. Appl. Math..

[47]  Xiang Chen,et al.  Variable Strength Interaction Testing with an Ant Colony System Approach , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[48]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[49]  Thang Trung Nguyen,et al.  Modified cuckoo search algorithm for short-term hydrothermal scheduling , 2015 .

[50]  Atif M. Memon,et al.  Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback , 2010, IEEE Transactions on Software Engineering.

[51]  Kamal Z. Zamli,et al.  A Test Generation Strategy For Variable-Strength and T-way Interaction , 2014 .

[52]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[53]  Myra B. Cohen,et al.  GUI Interaction Testing: Incorporating Event Context , 2011, IEEE Transactions on Software Engineering.

[54]  Rusli Abdullah,et al.  Design and implementation of a t-way test data generation strategy with automated execution tool support , 2011, Inf. Sci..

[55]  Myra B. Cohen,et al.  Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization , 2007, 2007 IEEE International Conference on Software Maintenance.

[56]  Xin-She Yang,et al.  Cuckoo Search via Lévy flights , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[57]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[58]  Hareton K. N. Leung,et al.  Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures , 2015, Inf. Softw. Technol..

[59]  D. Richard Kuhn,et al.  Combinatorial and Random Testing Effectiveness for a Grid Computer Simulator , 2009 .

[60]  Lucia Moura,et al.  Covering arrays avoiding forbidden edges , 2009, Theor. Comput. Sci..

[61]  Donald L. Kreher,et al.  On the state of strength‐three covering arrays , 2002 .

[62]  Phil McMinn,et al.  Search-based software test data generation: a survey: Research Articles , 2004 .

[63]  Xin-She Yang,et al.  Nature-Inspired Metaheuristic Algorithms: Second Edition , 2010 .

[64]  M. R. Brown,et al.  A Review of the Development and Applications of the Cuckoo Search Algorithm , 2013 .

[65]  A. Hartman.  Software and Hardware Testing Using Combinatorial Covering Suites , 2005 .

[66]  Myra B. Cohen,et al.  Towards incremental adaptive covering arrays , 2007, ESEC-FSE '07.

[67]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[68]  Xiangtao Li,et al.  Modified cuckoo search algorithm with self adaptive parameter method , 2015, Inf. Sci..

[69]  Myra B. Cohen,et al.  Moving Forward with Combinatorial Interaction Testing , 2014, Computer.

[70]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[71]  J. Czerwonka.  Pairwise Testing in Real World Practical Extensions to Test Case Generators , 2006 .

[72]  Soumen Maity,et al.  Improved test generation algorithms for pair-wise testing , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[73]  Xin-She Yang,et al.  Cuckoo search: recent advances and applications , 2013, Neural Computing and Applications.

[74]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[75]  Srinivas Nidhra,et al.  WHITE BOX TESTING TECHNIQUES –A LITERATURE REVIEW , 2012 .

[76]  D. Richard Kuhn,et al.  Combinatorial Testing for Software: an Adaptation of Design of Experiments , 2013 .