A multi-objective approach to search-based test data generation

There has been a considerable body of work on search-based test data generation for branch coverage. However, hitherto, there has been no work on multi-objective branch coverage. In many scenarios a single-objective formulation is unrealistic; testers will want to find test sets that meet several objectives simultaneously in order to maximize the value obtained from the inherently expensive process of running the test cases and examining the output they produce. This paper introduces multi-objective branch coverage.The paper presents results from a case study of the twin objectives of branch coverage and dynamic memory consumption for both real and synthetic programs. Several multi-objective evolutionary algorithms are applied. The results show that multi-objective evolutionary algorithms are suitable for this problem, and illustrates the way in which a Pareto optimal search can yield insights into the trade-offs between the two simultaneous objectives.

[1]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[2]  Mark Harman,et al.  Testability Transformation for Efficient Automated Test Data Search in the Presence of Nesting , 2005 .

[3]  McMinnPhil.  Search-based software test data generation: a survey , 2004 .

[4]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[5]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[6]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[7]  Zbigniew Michalewicz,et al.  Evolutionary Computation 1 , 2018 .

[8]  Irman Hermadi,et al.  Genetic algorithm based test data generator , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[9]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[10]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[11]  Carlos A. Coello Coello,et al.  Handling constraints using multiobjective optimization concepts , 2004 .

[12]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[13]  Nashat Mansour,et al.  Data Generation for Path Testing , 2004, Software Quality Journal.

[14]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[15]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[16]  James E. Baker,et al.  Reducing Bias and Inefficienry in the Selection Algorithm , 1987, ICGA.

[17]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[18]  Mark Harman,et al.  Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs , 2006, Comput. J..

[19]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[20]  Leonardo Bottaci.  Instrumenting Programs With Flag Variables For Test Data Search By Genetic Algorithms , 2002, GECCO.

[21]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999, Softw. Test. Verification Reliab..

[22]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[23]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[24]  Phil McMinn,et al.  IGUANA: Input Generation Using Automated Novel Algorithms. A Plug and Play Research Tool , 2007 .

[25]  A. Jefferson Offutt.  An integrated system for automatically generating test data , 1990, Systems Integration '90. Proceedings of the First International Conference on Systems Integration.

[26]  Lionel C. Briand,et al.  Stress testing real-time systems with genetic algorithms , 2005, GECCO '05.

[27]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[28]  André Baresel,et al.  Fitness Function Design To Improve Evolutionary Structural Testing , 2002, GECCO.

[29]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[30]  Marek Reformat,et al.  Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques , 2007, Empirical Software Engineering.

[31]  Paolo Toth,et al.  Knapsack Problems: Algorithms and Computer Implementations , 1990 .