It really does matter how you normalize the branch distance in search‐based software testing

The use of search algorithms for test data generation has seen many successful results. For structural criteria like branch coverage, heuristics have been designed to help the search. The most common heuristic is the use of approach level (usually represented with an integer) to reward test cases whose executions get close (in the control flow graph) to the target branch. To solve the constraints of the predicates in the control flow graph, the branch distance is commonly employed. These two measures are linearly combined. Since the approach level is more important, the branch distance is normalized, often in the range [0, 1]. In this paper, different types of normalizing functions are analyzed. The analyses show that the one that is usually employed in the literature has several flaws. The paper presents a different normalizing function that is very simple and does not suffer from these limitations. Empirical and analytical analyses are carried out to compare these two functions. In particular, their effect is studied on commonly used search algorithms, such as Hill Climbing, Simulated Annealing and Genetic Algorithms. Copyright © 2011 John Wiley & Sons, Ltd.

[1]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[2]  Mark Harman,et al.  Constructing multiple unique input/output sequences using metaheuristic optimisation techniques , 2005, IEE Proc. Softw..

[3]  Yuanyuan Zhang,et al.  Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications , 2009 .

[4]  Mark Harman,et al.  The species per path approach to SearchBased test data generation , 2006, ISSTA '06.

[5]  John Fulcher,et al.  Computational Intelligence: An Introduction , 2008, Computational Intelligence: A Compendium.

[6]  Andrea Arcuri,et al.  It Does Matter How You Normalise the Branch Distance in Search Based Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[7]  Nashat Mansour,et al.  Data Generation for Path Testing , 2004, Software Quality Journal.

[8]  Mark Harman,et al.  Empirical evaluation of a nesting testability transformation for evolutionary testing , 2009, TSEM.

[9]  John A. Clark,et al.  Using automated search to generate test data for matlab , 2009, GECCO.

[10]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[11]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[12]  Lionel C. Briand,et al.  Formal analysis of the effectiveness and predictability of random testing , 2010, ISSTA '10.

[13]  Sergiu M. Dascalu,et al.  Unit-level test adequacy criteria for visual dataflow languages and a testing methodology , 2008, TSEM.

[14]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[15]  Xin Yao,et al.  Search based software testing of object-oriented containers , 2008, Inf. Sci..

[16]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[17]  Pietro Simone Oliveto,et al.  Theoretical analysis of fitness-proportional selection: landscapes and efficiency , 2009, GECCO.

[18]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[19]  Hélène Waeselynck,et al.  Simulated annealing applied to test generation: landscape characterization and stopping criteria , 2007, Empirical Software Engineering.

[20]  Russ Bubley,et al.  Randomized algorithms , 2018, CSUR.

[21]  Rami Bahsoon,et al.  Empirical comparison of regression test selection algorithms , 2001, J. Syst. Softw..

[22]  X. Yao,et al.  Theoretical Runtime Analysis in Search Based Software Engineering , 2009 .

[23]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[24]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[25]  Andrea Arcuri,et al.  Theoretical Analysis of Local Search in Software Testing , 2009, SAGA.

[26]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[27]  Huai Liu,et al.  Adaptive random testing based on distribution metrics , 2009, J. Syst. Softw..

[28]  Gordon Fraser,et al.  It is Not the Length That Matters, It is How You Control It , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[29]  Stephen Swift,et al.  A Search-Based Approach for Automatic Test Generation from Extended Finite State Machine (EFSM) , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[30]  Marek Reformat,et al.  Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques , 2007, Empirical Software Engineering.

[31]  Per Kristian Lehre,et al.  Theoretical Runtime Analyses of Search Algorithms on the Test Data Generation for the Triangle Classification Problem , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[32]  Andrea Arcuri,et al.  Full Theoretical Runtime Analysis of Alternating Variable Method on the Triangle Classification Problem , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[33]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[34]  Andrea Arcuri,et al.  Insight knowledge in search based software testing , 2009, GECCO.

[35]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[36]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[37]  Günther Ruhe,et al.  Search Based Software Engineering , 2013, Lecture Notes in Computer Science.

[38]  Andrea Arcuri On Search Based Software Evolution , 2009, 2009 1st International Symposium on Search Based Software Engineering.