Analysing the fitness landscape of search-based software testing problems

Search-based software testing automatically derives test inputs for a software system with the goal of improving various criteria, such as branch coverage. In many cases, evolutionary algorithms are implemented to find near-optimal test suites for software systems. The result of the search is usually received without any indication of how successful the search has been. Fitness landscape characterisation can help understand the search process and its probability of success. In this study, we recorded the information content, negative slope coefficient and the number of improvements during the progress of a genetic algorithm within the EvoSuite framework. Correlating the metrics with the branch and method coverages and the fitness function values reveals that the problem formulation used in EvoSuite could be improved by revising the objective function. It also demonstrates that given the current formulation, the use of crossover has no benefits for the search as the most problematic landscape features are not the number of local optima but the presence of many plateaus.

[1]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[2]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[3]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999, Softw. Test. Verification Reliab..

[4]  Marouane Kessentini,et al.  Preference-Based Many-Objective Evolutionary Testing Generates Harder Test Cases for Autonomous Agents , 2013, SSBSE.

[5]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[6]  Julian Francis Miller,et al.  Information Characteristics and the Structure of Landscapes , 2000, Evolutionary Computation.

[7]  Gordon Fraser,et al.  Parameter tuning or default values? An empirical investigation in search-based software engineering , 2013, Empirical Software Engineering.

[8]  Vassilis Zissimopoulos,et al.  Autocorrelation Coefficient for the Graph Bipartitioning Problem , 1998, Theor. Comput. Sci..

[9]  Michael Affenzeller,et al.  A Comprehensive Survey on Fitness Landscape Analysis , 2012, Recent Advances in Intelligent Engineering Systems.

[10]  Irene Moser,et al.  Designing and characterising fitness landscapes with various operators , 2013, 2013 IEEE Congress on Evolutionary Computation.

[11]  Darrel C. Ince,et al.  The Automatic Generation of Test Data , 1987, Comput. J..

[12]  Scott Kirkpatrick,et al.  Optimization by Simmulated Annealing , 1983, Sci..

[13]  K. Kinnear.  Fitness landscapes and difficulty in genetic programming , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[14]  Weinberger,et al.  Local properties of Kauffman's N-k model: A tunably rugged energy landscape. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[15]  Thomas Bäck,et al.  An Empirical Study on GAs "Without Parameters" , 2000, PPSN.

[16]  David L. Spooner,et al.  Automatic Generation of Floating-Point Test Data , 1976, IEEE Transactions on Software Engineering.

[17]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[18]  Gordon Fraser,et al.  A Memetic Algorithm for whole test suite generation , 2015, J. Syst. Softw..

[19]  Phil Husbands,et al.  Fitness Landscapes and Evolvability , 2002, Evolutionary Computation.

[20]  Baowen Xu,et al.  Application of Genetic Algorithms in Software Testing , 2007 .

[21]  Irene Moser,et al.  Characterising fitness landscapes using predictive local search , 2013, GECCO '13 Companion.

[22]  Bryan F. Jones,et al.  Automatic structural testing using genetic algorithms , 1996, Softw. Eng. J..

[23]  Bernd Freisleben,et al.  Fitness landscape analysis and memetic algorithms for the quadratic assignment problem , 2000, IEEE Trans. Evol. Comput..

[24]  Jie Chen,et al.  Problem difficulty analysis for particle swarm optimization: deception and modality , 2009, GEC '09.

[25]  Sébastien Vérel,et al.  Negative Slope Coefficient: A Measure to Characterize Genetic Programming Fitness Landscapes , 2006, EuroGP.

[26]  Roy P. Pargas,et al.  Test-Data Generation Using Genetic Algorithms , 1999, Softw. Test. Verification Reliab..

[27]  Gordon Fraser,et al.  On Parameter Tuning in Search Based Software Engineering , 2011, SSBSE.

[28]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[29]  Marc Roper.  Computer aided software testing using genetic algorithms , 1997 .

[30]  Karl N. Levitt,et al.  SELECT—a formal system for testing and debugging programs by symbolic execution , 1975 .

[31]  Kate Smith-Miles,et al.  Measuring instance difficulty for combinatorial optimization problems , 2012, Comput. Oper. Res..

[32]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[33]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[34]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[35]  Lionel C. Briand,et al.  Search-based automated testing of continuous controllers: Framework, tool support, and case studies , 2015, Inf. Softw. Technol..

[36]  Chengying Mao,et al.  Adapting ant colony optimization to generate test data for software structural testing , 2015, Swarm Evol. Comput..

[37]  Alessandra Gorla,et al.  Search-based data-flow test generation , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[38]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[39]  Lars Grunske,et al.  Test data generation with a Kalman filter-based adaptive genetic algorithm , 2015, J. Syst. Softw..

[40]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[41]  P. Stadler.  Landscapes and their correlation functions , 1996 .

[42]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[43]  HarmanMark,et al.  Using formal specifications to support testing , 2009 .

[44]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[45]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[46]  Bruno Marre,et al.  PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis , 2005, EDCC.

[47]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[48]  E. Weinberger,et al.  Correlated and uncorrelated fitness landscapes and how to tell the difference , 1990, Biological Cybernetics.