Empirical evaluation of a nesting testability transformation for evolutionary testing

Evolutionary testing is an approach to automating test data generation that uses an evolutionary algorithm to search a test object's input domain for test data. Nested predicates can cause problems for evolutionary testing, because information needed for guiding the search only becomes available as each nested conditional is satisfied. This means that the search process can overfit to early information, making it harder, and sometimes near impossible, to satisfy constraints that only become apparent later in the search. The article presents a testability transformation that allows the evaluation of all nested conditionals at once. Two empirical studies are presented. The first study shows that the form of nesting handled is prevalent in practice. The second study shows how the approach improves evolutionary test data generation.

[1]  Mark Harman,et al.  Improving Evolutionary Testing By Flag Removal , 2002, GECCO.

[2]  André Baresel,et al.  Fitness Function Design To Improve Evolutionary Structural Testing , 2002, GECCO.

[3]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[4]  Mark Harman,et al.  Testability Transformation for Efficient Automated Test Data Search in the Presence of Nesting , 2005 .

[5]  Mark Harman,et al.  The impact of input domain reduction on search-based test data generation , 2007, ESEC-FSE '07.

[6]  John A. Clark,et al.  Automated test‐data generation for exception conditions , 2000, Softw. Pract. Exp..

[7]  R. Gupta,et al.  Data dependence based testability transformation in automated test generation , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[8]  John A. Clark,et al.  The Way Forward for Unifying Dynamic Test Case Generation: The Optimisation-based Approach , 1998 .

[9]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[10]  Nigel James Tracey,et al.  A search-based automated test-data generation framework for safety-critical software , 2000 .

[11]  Peter P. Puschner,et al.  Testing the results of static worst-case execution-time analysis , 1998, Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No.98CB36279).

[12]  Karl N. Levitt,et al.  SELECT—a formal system for testing and debugging programs by symbolic execution , 1975 .

[13]  Bogdan Korel,et al.  Assertion-oriented automated test data generation , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[14]  Mark Harman,et al.  Branch-Coverage Testability Transformation for Unstructured Programs , 2005, Comput. J..

[15]  B. F. Jones,et al.  The Automatic Generation Of Software Test Data Sets Using Adaptive Search Techniques , 1970 .

[16]  Phil McMinn,et al.  Evolutionary Testing Using an Extended Chaining Approach , 2006, Evolutionary Computation.

[17]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[18]  Bogdan Korel,et al.  Dynamic method for software test data generation , 1992, Softw. Test. Verification Reliab..

[19]  Bogdan Korel,et al.  The chaining approach for software test data generation , 1996, TSEM.

[20]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999 .

[21]  Mark Harman,et al.  A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation , 2007, ISSTA '07.

[22]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[23]  John A. Clark,et al.  Automated test‐data generation for exception conditions , 2000 .

[24]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[25]  L. Darrell Whitley,et al.  An overview of evolutionary algorithms: practical issues and common pitfalls , 2001, Inf. Softw. Technol..

[26]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[27]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[28]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[29]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[30]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[31]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[32]  Roy P. Pargas,et al.  Test-Data Generation Using Genetic Algorithms , 1999, Softw. Test. Verification Reliab..

[33]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[34]  Mark Harman,et al.  Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach , 2004, ISSTA '04.

[35]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[36]  Matthias Grochtmann,et al.  Systematic Testing of Real-Time Systems , 2000 .

[37]  André Baresel,et al.  Evolutionary Testing of Flag Conditions , 2003, GECCO.

[38]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[39]  Matthias Grochtmann,et al.  Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing , 1998, Real-Time Systems.