Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach

Evolutionary testing is an effective technique for automatically generating good quality test data. However, for structural testing, the technique degenerates to random testing in the presence of flag variables, which also present problems for other automated test data generation techniques. Previous work on the flag problem does not address flags assigned in loops.This paper introduces a testability transformation that transforms programs with loop--assigned flags so that existing genetic approaches can be successfully applied. It then presents empirical data demonstrating the effectiveness of the transformation. Untransformed, the genetic algorithm flounders and is unable to find a solution. Two transformations are considered. The first allows the search to find a solution. The second reduces the time taken by an order of magnitude and, more importantly, reduces the slope of the cost increase; thus, greatly increasing the complexity of the problem to which the genetic algorithm can be applied. The paper also presents a second empirical study showing that loop--assigned flags are prevalent in real world code. They account for just under 11% of all flags.

[1]  Joachim Wegener,et al.  Testing real-time systems using genetic algorithms , 1997, Software Quality Journal.

[2]  A. Jefferson Offutt.  An integrated system for automatically generating test data , 1990, Systems Integration '90. Proceedings of the First International Conference on Systems Integration.

[3]  Hartmut Pohlheim,et al.  Genetic and evolutionary algorithm toolbox for use with matlab , 1994 .

[4]  John Darlington,et al.  A Transformation System for Developing Recursive Programs , 1977, J. ACM.

[5]  John A. Clark,et al.  Formulating software engineering as a search problem , 2003, IEE Proc. Softw..

[6]  P. David Coward.  Symbolic execution systems-a review , 1988, Softw. Eng. J..

[7]  André Baresel,et al.  Evolutionary Testing of Flag Conditions , 2003, GECCO.

[8]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[9]  Mark Harman,et al.  An Empirical Investigation of the Influence of a Type of Side Effects on Program Comprehension , 2003, IEEE Trans. Software Eng..

[10]  J. Voas,et al.  Software Testability: The New Verification , 1995, IEEE Softw..

[11]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[12]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[13]  A. Jefferson Offutt,et al.  Experimental results from an automatic test case generator , 1993, TSEM.

[14]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[15]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[16]  Joachim Wegener,et al.  A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints , 2004, Real-Time Systems.

[17]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[18]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[19]  Joachim Wegener,et al.  Testing the temporal behavior of real-time software modules using extended evolutionary algorithms , 1999 .

[20]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[21]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[22]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[23]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[24]  Roy P. Pargas,et al.  Test-Data Generation Using Genetic Algorithms , 1999, Softw. Test. Verification Reliab..

[25]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[26]  Peter P. Puschner,et al.  Testing the results of static worst-case execution-time analysis , 1998, Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No.98CB36279).

[27]  Martin S. Feather,et al.  A System for Assisting Program Transformation , 1982, TOPL.

[28]  Mark David Weiser,et al.  Program slices: formal, psychological, and practical investigations of an automatic program abstraction method , 1979 .

[29]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[30]  Leonardo Bottaci.  Instrumenting Programs With Flag Variables For Test Data Search By Genetic Algorithms , 2002, GECCO.

[31]  Bryan F. Jones,et al.  A Strategy for Using Genetic Algorithms to Automate Branch and Fault-Based Testing , 1998, Comput. J..

[32]  Helmuth Partsch,et al.  Specification and Transformation of Programs - A Formal Approach to Software Development , 1990, Texts and Monographs in Computer Science.

[33]  Zohar Manna,et al.  The evolution of programs: a system for automatic program modification , 1977, POPL '77.

[34]  Matthias Grochtmann,et al.  Systematic Testing of Real-Time Systems , 2000 .

[35]  John J. Grefenstette,et al.  Test and evaluation by genetic algorithms , 1993, IEEE Expert.

[36]  Frank Tip,et al.  A survey of program slicing techniques , 1994, J. Program. Lang..

[37]  John A. Clark,et al.  The Way Forward for Unifying Dynamic Test Case Generation: The Optimisation-based Approach , 1998 .

[38]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[39]  André Baresel,et al.  Fitness Function Design To Improve Evolutionary Structural Testing , 2002, GECCO.