An Empirical Study of Pairwise Test Set Generation Using a Genetic Algorithm

Pairwise test set generation is the process of producing a subset of all possible test case inputs to a system in situations where exhaustive testing is not possible or is prohibitively expensive. For a given system under test with a set of input parameters where each parameter can take on one of a discrete set of values, a pairwise test set consists of a collection of vectors which capture all possible combinations of pairs of parameter values. Generating pairwise test sets with a minimal size has been shown to be an NP-complete problem, and several deterministic generation algorithms have been published. This paper describes the results of an investigation of pairwise test set generation using a genetic algorithm. The genetic algorithm approach produced pairwise test sets with comparable or smaller (better) size compared with published results for deterministic algorithms for 39 out of 40 benchmark problems. However, the genetic algorithm test set generation technique required significantly longer processing time in all cases. The results illustrate that generation of pairwise test sets using a genetic algorithm is possible, and suggest that the technique may be both practical and useful in certain software testing situations.

[1]  Moataz A. Ahmed,et al.  Pair-wise test coverage using genetic algorithms , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[2]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[3]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[4]  Patrick J. Schroeder,et al.  Pairwise Testing : A Best Practice That Isn ’ , 2005 .

[5]  Robert L. Probert,et al.  A practical strategy for testing pair-wise coverage of network interfaces , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[6]  Chen Zengqiang,et al.  Improved crossover strategy of genetic algorithms and analysis of its performance , 2000, Proceedings of the 3rd World Congress on Intelligent Control and Automation (Cat. No.00EX393).

[7]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[8]  Tzung-Pei Hong,et al.  A dynamic mutation genetic algorithm , 1996, 1996 IEEE International Conference on Systems, Man and Cybernetics. Information Intelligence and Systems (Cat. No.96CH35929).

[9]  Jun Yan,et al.  Backtracking Algorithms and Search Heuristics to Generate Test Suites for Combinatorial Testing , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[10]  Geraldo Robson Mateus,et al.  Hierarchical tournament selection genetic algorithm for the vehicle routing problem with time windows , 2004, Fourth International Conference on Hybrid Intelligent Systems (HIS'04).

[11]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[12]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[13]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .