Input Domain Reduction through Irrelevant Variable Removal and Its Effect on Local, Global, and Hybrid Search-Based Structural Test Data Generation

Search-Based Test Data Generation reformulates testing goals as fitness functions so that test input generation can be automated by some chosen search-based optimization algorithm. The optimization algorithm searches the space of potential inputs, seeking those that are “fit for purpose,” guided by the fitness function. The search space of potential inputs can be very large, even for very small systems under test. Its size is, of course, a key determining factor affecting the performance of any search-based approach. However, despite the large volume of work on Search-Based Software Testing, the literature contains little that concerns the performance impact of search space reduction. This paper proposes a static dependence analysis derived from program slicing that can be used to support search space reduction. The paper presents both a theoretical and empirical analysis of the application of this approach to open source and industrial production code. The results provide evidence to support the claim that input domain reduction has a significant effect on the performance of local, global, and hybrid search, while a purely random search is unaffected.

[1]  J. Voas,et al.  Software Testability: The New Verification , 1995, IEEE Softw..

[2]  Frank Schmiedle,et al.  Exact Routing with Search Space Reduction , 2003, IEEE Trans. Computers.

[3]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[4]  Mark Harman,et al.  The species per path approach to SearchBased test data generation , 2006, ISSTA '06.

[5]  Mark Harman,et al.  The impact of input domain reduction on search-based test data generation , 2007, ESEC-FSE '07.

[6]  Outi Räihä,et al.  A survey on search-based software design , 2010, Comput. Sci. Rev..

[7]  Oliver Buehler,et al.  Evolutionary Functional Testing of an Automated Parking System , 2003 .

[8]  Mark Harman,et al.  Reducing qualitative human oracle costs associated with automatically generated test data , 2010, STOV '10.

[9]  Francisco Fernández de Vega,et al.  Test Case Evaluation and Input Domain Reduction strategies for the Evolutionary Testing of Object-Oriented software , 2009, Inf. Softw. Technol..

[10]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[11]  Andrea Arcuri,et al.  It Does Matter How You Normalise the Branch Distance in Search Based Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[12]  Mark Harman,et al.  n empirical investigation into branch coverage for C programs using CUTE and USTIN , 2010 .

[13]  Gregor Snelting,et al.  Validation of measurement software as an application of slicing and constraint solving , 1998, Inf. Softw. Technol..

[14]  Mark Harman,et al.  An empirical study of static program slice size , 2007, TSEM.

[15]  McMinnPhil Search-based software test data generation: a survey , 2004 .

[16]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[17]  Petr Posík Estimation of Distribution Algorithms , 2006 .

[18]  Lionel C. Briand,et al.  Using genetic algorithms and coupling measures to devise optimal integration test orders , 2002, SEKE '02.

[19]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[20]  Joachim Wegener,et al.  Evaluating Evolutionary Testability with Software-Measurements , 2004, GECCO.

[21]  Mark Harman,et al.  VADA: a transformation-based system for variable dependence analysis , 2002, Proceedings. Second IEEE International Workshop on Source Code Analysis and Manipulation.

[22]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[23]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[24]  Nashat Mansour,et al.  Data Generation for Path Testing , 2004, Software Quality Journal.

[25]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[26]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[27]  Mark Harman,et al.  Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[28]  José Antonio Lozano,et al.  DYNAMIC SEARCH SPACE TRANSFORMATIONS FOR SOFTWARE TEST DATA GENERATION , 2008, Comput. Intell..

[29]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[30]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[31]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[32]  Matthias Grochtmann,et al.  Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing , 1998, Real-Time Systems.

[33]  Lars Ole Andersen,et al.  Program Analysis and Specialization for the C Programming Language , 2005 .

[34]  Lionel C. Briand,et al.  Formal analysis of the effectiveness and predictability of random testing , 2010, ISSTA '10.

[35]  Pedro Larrañaga,et al.  Estimation of Distribution Algorithms , 2002, Genetic Algorithms and Evolutionary Computation.

[36]  Yuanyuan Zhang,et al.  Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications , 2009 .

[37]  David W. Binkley,et al.  Interprocedural slicing using dependence graphs , 1990, TOPL.

[38]  Patrick Cousot,et al.  Abstract Interpretation Frameworks , 1992, J. Log. Comput..

[39]  James E. Baker,et al.  Reducing Bias and Inefficienry in the Selection Algorithm , 1987, ICGA.

[40]  Roy P. Pargas,et al.  Test-Data Generation Using Genetic Algorithms , 1999, Softw. Test. Verification Reliab..

[41]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[42]  Mark Harman,et al.  Automated test data generation for aspect-oriented programs , 2009, AOSD '09.

[43]  Andrea Arcuri,et al.  Theoretical Analysis of Local Search in Software Testing , 2009, SAGA.

[44]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[45]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[46]  Bjarne Steensgaard,et al.  Points-to analysis in almost linear time , 1996, POPL '96.

[47]  Stephen F. Smith,et al.  Improving Genetic Algorithms by Search Space Reductions (with Applications to Flow Shop Scheduling) , 1999, GECCO.

[48]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[49]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.