A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation

Evolutionary testing has been widely studied as a technique for automating the process of test case generation. However, to date, there has been no theoretical examination of when and why it works. Furthermore, the empirical evidence for the effectiveness of evolutionary testing consists largely of small scale laboratory studies. This paper presents a first theoretical analysis of the scenarios in which evolutionary algorithms are suitable for structural test case generation. The theory is backed up by an empirical study that considers real world programs, the search spaces of which are several orders of magnitude larger than those previously considered.

[1]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[2]  Mark Harman,et al.  The species per path approach to SearchBased test data generation , 2006, ISSTA '06.

[3]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[4]  Nashat Mansour,et al.  Data Generation for Path Testing , 2004, Software Quality Journal.

[5]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[6]  McMinnPhil.  Search-based software test data generation: a survey , 2004 .

[7]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[8]  André Baresel,et al.  Fitness Function Design To Improve Evolutionary Structural Testing , 2002, GECCO.

[9]  James E. Baker,et al.  Reducing Bias and Inefficienry in the Selection Algorithm , 1987, ICGA.

[10]  Zbigniew Michalewicz,et al.  Evolutionary Computation 1 , 2018 .

[11]  Mark Harman,et al.  Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs , 2006, Comput. J..

[12]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[13]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[14]  Marek Reformat,et al.  Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques , 2007, Empirical Software Engineering.

[15]  Leonardo Bottaci.  Instrumenting Programs With Flag Variables For Test Data Search By Genetic Algorithms , 2002, GECCO.

[16]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[17]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999, Softw. Test. Verification Reliab..

[18]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[19]  Phil McMinn,et al.  Evolutionary Testing Using an Extended Chaining Approach , 2006, Evolutionary Computation.

[20]  Phil McMinn,et al.  IGUANA: Input Generation Using Automated Novel Algorithms. A Plug and Play Research Tool , 2007 .

[21]  André Baresel,et al.  Evolutionary Testing of Flag Conditions , 2003, GECCO.

[22]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[23]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[24]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[25]  Melanie Mitchell,et al.  The royal road for genetic algorithms: Fitness landscapes and GA performance , 1991 .

[26]  Lionel C. Briand,et al.  Stress testing real-time systems with genetic algorithms , 2005, GECCO '05.

[27]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[28]  Colin R. Reeves,et al.  Genetic Algorithms: Principles and Perspectives: A Guide to Ga Theory , 2002 .