Observations in using parallel and sequential evolutionary algorithms for automatic software testing

In this paper we analyze the application of parallel and sequential evolutionary algorithms (EAs) to the automatic test data generation problem. The problem consists of automatically creating a set of input data to test a program. This is a fundamental step in software development and a time consuming task in existing software companies. Canonical sequential EAs have been used in the past for this task. We explore here the use of parallel EAs. Evidence of greater efficiency, larger diversity maintenance, additional availability of memory/CPU, and multi-solution capabilities of the parallel approach, reinforce the importance of the advances in research with these algorithms. We describe in this work how canonical genetic algorithms (GAs) and evolutionary strategies (ESs) can help in software testing, and what the advantages are (if any) of using decentralized populations in these techniques. In addition, we study the influence of some parameters of the proposed test data generator in the results. For the experiments we use a large benchmark composed of twelve programs that includes fundamental algorithms in computer science.

[1]  Phil McMinn,et al.  The State Problem for Evolutionary Testing , 2003, GECCO.

[2]  Alberto Lluch-Lafuente.  Symmetry reduction and heuristic search for error detection in model checking , 2003 .

[3]  Edmund M. Clarke,et al.  Symbolic model checking for sequential circuit verification , 1993, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[4]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching-Time Temporal Logic , 1981, Logic of Programs.

[5]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[6]  Nashat Mansour,et al.  Data Generation for Path Testing , 2004, Software Quality Journal.

[7]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[8]  Alex Groce,et al.  Heuristics for model checking Java programs , 2004, International Journal on Software Tools for Technology Transfer.

[9]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[10]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[11]  C. A. R. HOARE,et al.  An axiomatic basis for computer programming , 1969, CACM.

[12]  W. Vent,et al.  Rechenberg, Ingo, Evolutionsstrategie — Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. 170 S. mit 36 Abb. Frommann‐Holzboog‐Verlag. Stuttgart 1973. Broschiert , 1975 .

[13]  David L. Levine,et al.  Users guide to the PGAPack parallel genetic algorithm library , 1995 .

[14]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[15]  L. Darrell Whitley,et al.  Serial and Parallel Genetic Algorithms as Function Optimizers , 1993, ICGA.

[16]  H. Dieter Rombach.  Proceedings of the 18th international conference on Software engineering , 1996, ICSE 1996.

[17]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[18]  Vidroha Debroy,et al.  Genetic Programming , 1998, Lecture Notes in Computer Science.

[19]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[20]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[21]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[22]  José Antonio Lozano,et al.  Scatter Search in software testing, comparison and collaboration with Estimation of Distribution Algorithms , 2006, Eur. J. Oper. Res..

[23]  Stephan Merz,et al.  Model Checking , 2000 .

[24]  Theodore C. Belding,et al.  The Distributed Genetic Algorithm Revisited , 1995, ICGA.

[25]  Timo Mantere,et al.  Evolutionary software engineering, a review , 2005, Appl. Soft Comput..

[26]  Harmen-Hinrich Sthamer,et al.  The automatic generation of software test data using genetic algorithms , 1995 .

[27]  J. David Schaffer,et al.  Proceedings of the third international conference on Genetic algorithms , 1989 .

[28]  César Muñoz,et al.  An Overview of SAL , 2000 .

[29]  David B. Fogel,et al.  Evolution-ary Computation 1: Basic Algorithms and Operators , 2000 .

[30]  Enrique Alba,et al.  Parallelism and evolutionary algorithms , 2002, IEEE Trans. Evol. Comput..

[31]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[32]  Zbigniew Michalewicz,et al.  Evolutionary Computation 1 , 2018 .

[33]  David L. Dill,et al.  Java model checking , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[34]  Baowen Xu,et al.  Application of Genetic Algorithms in Software Testing , 2007 .

[35]  Shumeet Baluja,et al.  Structure and Performance of Fine-Grain Parallelism in Genetic Search , 1993, ICGA.

[36]  Bogdan Korel,et al.  Assertion-oriented automated test data generation , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[37]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[38]  Carlos Urias Munoz,et al.  Automatic Generation of Random Self-Checking Test Cases , 1983, IBM Syst. J..

[39]  Nigel James Tracey,et al.  A search-based automated test-data generation framework for safety-critical software , 2000 .

[40]  James C. Corbett,et al.  Bandera: extracting finite-state models from Java source code , 2000, ICSE.

[41]  Edsger W. Dijkstra,et al.  Predicate Calculus and Program Semantics , 1989, Texts and Monographs in Computer Science.

[42]  Enrique Alba,et al.  Parallel Metaheuristics: A New Class of Algorithms , 2005 .

[43]  Enrique Alba,et al.  Software Testing with Evolutionary Strategies , 2005, RISE.

[44]  Joachim Wegener,et al.  Evolutionary Functional Testing of a Vehicle Brake Assistant System , 2005 .

[45]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[46]  John F. Passafiume,et al.  Software testing and evaluation , 1987 .

[47]  Reiko Tanese,et al.  Distributed Genetic Algorithms , 1989, ICGA.

[48]  Oliver Buehler,et al.  Evolutionary Functional Testing of an Automated Parking System , 2003 .

[49]  Enrique Alba,et al.  A survey of parallel distributed genetic algorithms , 1999, Complex..

[50]  David S. Alberts,et al.  The economics of software quality assurance , 1976, AFIPS '76.

[51]  R. Sagarna,et al.  Variable search space for software testing , 2003, International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003.

[52]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[53]  Edsger W. Dijkstra,et al.  A Discipline of Programming , 1976 .

[54]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[55]  Erwin Engeler,et al.  Logic of Programs, Workshop , 1981 .

[56]  Stefan Edelkamp,et al.  Partial Order Reduction in Directed Model Checking , 2002, SPIN.

[57]  A. Jefferson Offutt,et al.  An integrated automatic test data generation system , 1991, J. Syst. Integr..

[58]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[59]  Eugenia Díaz,et al.  Automated software testing using a metaheuristic technique based on Tabu search , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[60]  Joachim Wegener,et al.  Using Evolutionary Testing to improve Efficiency and Quality in Software Testing , 2002 .

[61]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[62]  Lawrence J. Fogel,et al.  Intelligence Through Simulated Evolution: Forty Years of Evolutionary Programming , 1999 .

[63]  John R. Koza,et al.  Genetic programming (videotape): the movie , 1992 .

[64]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[65]  Enrique Alba,et al.  Gaining New Fields of Application for OOP : the Parallel Evolutionary Algorithm Case , 2001 .

[66]  Robert G. Reynolds,et al.  Knowledge-based software testing agent using evolutionary learning with cultural algorithms , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).