Runtime analysis of the (1 + 1) EA on computing unique input output sequences

Computing unique input output (UIO) sequences is a fundamental and hard problem in conformance testing of finite state machines (FSM). Previous experimental research has shown that evolutionary algorithms (EAs) can be applied successfully to find UIOs for some FSMs. However, before EAs can be recommended as a practical technique for computing UIOs, it is necessary to better understand the potential and limitations of these algorithms on this problem. In particular, more research is needed in determining for what instance classes of the problem EAs are feasible, and for what instance classes EAs are provably better than random search strategies. This paper presents rigorous theoretical and numerical analyses of the runtime of the (1+1) EA and random search on several selected instance classes of this problem. The theoretical analysis shows firstly, that there are instance classes where the EA is efficient, while random testing fails completely. Secondly, an instance class that is difficult for both random testing and the EA is presented. Finally, a parametrised instance class with tunable difficulty is presented. The numerical study estimates the constants in the asymptotic expressions obtained in the theoretical analysis, and the variability of the runtime. The numerical results fit well with the theoretical results, even for small problem instance sizes. Together, these results provide a first theoretical characterisation of the potential and limitations of the (1+1) EA on the problem of computing UIOs.

[1]  Mark Harman,et al.  Computing Unique Input/Output Sequences Using Genetic Algorithms , 2003, FATES.

[2]  Frank Neumann,et al.  Rigorous analyses of simple diversity mechanisms , 2007, GECCO '07.

[3]  Mark Harman,et al.  Heuristics for fault diagnosis when testing from finite state machines: Research Articles , 2007 .

[4]  Benjamin Doerr,et al.  Faster Evolutionary Algorithms by Superior Graph Representation , 2007, 2007 IEEE Symposium on Foundations of Computational Intelligence.

[5]  Alexandre Petrenko,et al.  Protocol testing: review of methods and relevance for software testing , 1994, ISSTA '94.

[6]  Mark Harman,et al.  Constructing multiple unique input/output sequences using metaheuristic optimisation techniques , 2005, IEE Proc. Softw..

[7]  Stefan Droste,et al.  A rigorous analysis of the compact genetic algorithm for linear functions , 2006, Natural Computing.

[8]  Mark Harman,et al.  Heuristics for fault diagnosis when testing from finite state machines , 2007, Softw. Test. Verification Reliab..

[9]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[10]  Ingo Wegener,et al.  Randomized local search, evolutionary algorithms, and the minimum spanning tree problem , 2004, Theor. Comput. Sci..

[11]  Qiang Guo,et al.  Improving fault coverage and minimising the cost of fault identification when testing from finite state machines , 2006 .

[12]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[13]  Mark Harman,et al.  Generating feasible input sequences for extended finite state machines (EFSMs) using genetic algorithms , 2005, GECCO '05.

[14]  Per Kristian Lehre,et al.  Runtime analysis of (1+l) EA on computing unique input output sequences , 2007, 2007 IEEE Congress on Evolutionary Computation.

[15]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[16]  Ingo Wegener,et al.  Fitness Landscapes Based on Sorting and Shortest Paths Problems , 2002, PPSN.

[17]  XI FachbereichInformatik.  Finite Markov Chain Results in Evolutionary Computation: a Tour D'horizon , 1998 .

[18]  Clifford Stein,et al.  Introduction to Algorithms, 2nd edition. , 2001 .

[19]  Oliver Giel,et al.  Zur Analyse von randomisierten Suchheuristiken und Online-Heuristiken , 2005 .

[20]  Deepinder P. Sidhu,et al.  Formal Methods for Protocol Testing: A Detailed Study , 1989, IEEE Trans. Software Eng..

[21]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[22]  Ingo Wegener,et al.  Real royal road functions--where crossover provably is essential , 2001, Discret. Appl. Math..

[23]  Xin Yao,et al.  Time complexity of evolutionary algorithms for combinatorial optimization: A decade of results , 2007, Int. J. Autom. Comput..

[24]  Per Kristian Lehre,et al.  On the effect of populations in evolutionary multi-objective optimization , 2006, GECCO.

[25]  Niraj K. Jha,et al.  Switching and Finite Automata Theory , 2010 .

[26]  Xin Yao,et al.  A study of drift analysis for estimating computation time of evolutionary algorithms , 2004, Natural Computing.

[27]  Karnig Derderian,et al.  Automated test sequence generation for finite state machines using genetic algorithms , 2006 .

[28]  Per Kristian Lehre,et al.  Theoretical Runtime Analyses of Search Algorithms on the Test Data Generation for the Triangle Classification Problem , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[29]  Thomas Jansen,et al.  Optimization with randomized search heuristics - the (A)NFL theorem, realistic scenarios, and difficult functions , 2002, Theor. Comput. Sci..

[30]  Russ Bubley,et al.  Randomized algorithms , 1995, CSUR.

[31]  Thomas Jansen,et al.  On the brittleness of evolutionary algorithms , 2007, FOGA'07.

[32]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[33]  Pietro Simone Oliveto,et al.  Design and Management of Complex Technical Processes and Systems by Means of Computational Intelligence Methods Theoretical Analysis of Diversity Mechanisms for Global Exploration Theoretical Analysis of Diversity Mechanisms for Global Exploration , 2022 .

[34]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[35]  Thomas Jansen,et al.  On the analysis of the (1+1) evolutionary algorithm , 2002, Theor. Comput. Sci..

[36]  Edward A. Feigenbaum,et al.  Switching and Finite Automata Theory: Computer Science Series , 1990 .

[37]  Heinz Mühlenbein,et al.  How Genetic Algorithms Really Work: Mutation and Hillclimbing , 1992, PPSN.

[38]  Andrea Arcuri,et al.  Theoretical Analysis of Local Search in Software Testing , 2009, SAGA.

[39]  Carsten Witt,et al.  Population size versus runtime of a simple evolutionary algorithm , 2008, Theor. Comput. Sci..

[40]  Frank Neumann,et al.  Randomized Local Search, Evolutionary Algorithms, and the Minimum Spanning Tree Problem , 2004, GECCO.

[41]  Per Kristian Lehre,et al.  Runtime analysis of search heuristics on software engineering problems , 2009, Frontiers of Computer Science in China.

[42]  Ingo Wegener,et al.  Real royal road functions for constant population size , 2003, Theor. Comput. Sci..

[43]  Mark Harman,et al.  A theoretical & empirical analysis of evolutionary testing and hill climbing for structural test data generation , 2007, ISSTA '07.

[44]  Mark Harman,et al.  Improving test quality using robust unique input/output circuit sequences (UIOCs) , 2006, Inf. Softw. Technol..

[45]  Schloss Birlinghoven,et al.  How Genetic Algorithms Really Work I.mutation and Hillclimbing , 2022 .

[46]  Mark Harman,et al.  Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs , 2006, Comput. J..