Using genetic algorithms to generate test sequences for complex timed systems

The generation of test data for state-based specifications is a computationally expensive process. This problem is magnified if we consider that time constraints have to be taken into account to govern the transitions of the studied system. The main goal of this paper is to introduce a complete methodology, supported by tools, that addresses this issue by representing the test data generation problem as an optimization problem. We use heuristics to generate test cases. In order to assess the suitability of our approach we consider two different case studies: a communication protocol and the scientific application BIPS3D. We give details concerning how the test case generation problem can be presented as a search problem and automated. Genetic algorithms (GAs) and random search are used to generate test data and evaluate the approach. GAs outperform random search and seem to scale well as the problem size increases. It is worth to mention that we use a very simple fitness function that can be easily adapted to be used with other evolutionary search techniques.

[1]  Mark Harman,et al.  Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs , 2006, Comput. J..

[2]  Krishnaiyan Thulasiraman,et al.  Context independent unique state identification sequences for testing communication protocols modelled as extended finite state machines , 2003, Comput. Commun..

[3]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[4]  Robert M. Hierons,et al.  Generating Feasible Transition Paths for Testing from an Extended Finite State Machine (EFSM) , 2009, ICST.

[5]  Mercedes G. Merayo,et al.  Formal testing from timed finite state machines , 2008, Comput. Networks.

[6]  Robert M. Hierons,et al.  Creating adaptive sequences with genetic algorithms to reach a certain state in a non-deterministic FSM , 2011, 2011 IEEE Symposium on Artificial Life (ALIFE).

[7]  Mercedes G. Merayo,et al.  Generation of optimal finite test suites for timed systems , 2007, First Joint IEEE/IFIP Symposium on Theoretical Aspects of Software Engineering (TASE '07).

[8]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[9]  Robert M. Hierons,et al.  A Case Study on the Use of Genetic Algorithms to Generate Test Cases for Temporal Systems , 2011, IWANN.

[10]  Robert M. Hierons,et al.  An integrated search-based approach for automatic testing from extended finite state machine (EFSM) models , 2011, Inf. Softw. Technol..

[11]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[12]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[13]  M. Ümit Uyar,et al.  Finite State Machine Based Formal Methods in Protocol Conformance Testing: From Theory to Implementation , 1991, Comput. Networks ISDN Syst..

[14]  Alexandre Petrenko,et al.  Confirming configurations in EFSM testing , 2004, IEEE Transactions on Software Engineering.

[15]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[16]  Robert M. Hierons,et al.  Aiding Test Case Generation in Temporally Constrained State Based Systems Using Genetic Algorithms , 2009, IWANN.

[17]  Lalit M. Patnaik,et al.  Genetic algorithms: a survey , 1994, Computer.

[18]  Robert M. Hierons,et al.  Optimizing the length of checking sequences , 2006, IEEE Transactions on Computers.

[19]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[20]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[21]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[22]  Robert M. Hierons,et al.  Experimental Comparison of Different Techniques to Generate Adaptive Sequences , 2011, IWANN.

[23]  Robert M. Hierons,et al.  Testing from a stochastic timed system with a fault model , 2009, J. Log. Algebraic Methods Program..

[24]  Karnig Derderian,et al.  Automated test sequence generation for finite state machines using genetic algorithms , 2006 .

[25]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[26]  Mark Harman,et al.  Formal methods and testing: an outcome of the FORTEST network , 2008 .

[27]  Eric Armengaud,et al.  Towards Cross-Domains Model-Based Safety Process, Methods and Tools for Critical Embedded Systems: The CESAR Approach , 2011, SAFECOMP.

[28]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[29]  Robert M. Hierons,et al.  Testing timed systems modeled by Stream X-machines , 2011, Software & Systems Modeling.

[30]  Bryan F. Jones,et al.  A Strategy for Using Genetic Algorithms to Automate Branch and Fault-Based Testing , 1998, Comput. J..

[31]  Mercedes G. Merayo,et al.  Extending EFSMs to Specify and Test Timed Systems with Action Durations and Time-Outs , 2008, IEEE Transactions on Computers.

[32]  Jesús Carretero,et al.  Analyzing Scalable High-Performance I/O Architectures , 2008, PDPTA.

[33]  Alexandre Petrenko,et al.  Fault Model-Driven Test Derivation from Finite State Models: Annotated Bibliography , 2000, MOVEP.

[34]  Richard Torkar,et al.  Search-based software testing and test data generation for a dynamic programming language , 2011, GECCO '11.

[35]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[36]  Mark Harman,et al.  Input Sequence Generation for Testing of Communicating Finite State Machines (CFSMs) , 2004, GECCO.

[37]  Manuel Núñez,et al.  Combining Genetic Algorithms and Mutation Testing to Generate Test Sequences , 2009, IWANN.

[38]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[39]  M. Ümit Uyar,et al.  A method enabling feasible conformance test sequence generation for EFSM models , 2004, IEEE Transactions on Computers.

[40]  Mark Harman,et al.  Estimating the feasibility of transition paths in extended finite state machines , 2010, Automated Software Engineering.

[41]  Zbigniew Michalewicz,et al.  Genetic algorithms + data structures = evolution programs (3rd ed.) , 1996 .

[42]  Tiankai Tu,et al.  High Resolution Forward And Inverse Earthquake Modeling on Terascale Computers , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[43]  L. D. Whitley A free lunch proof for Gray versus Binary encodings , 1999 .

[44]  Mercedes G. Merayo,et al.  A formal framework to test soft and hard deadlines in timed systems , 2012, Softw. Test. Verification Reliab..

[45]  F. C. Hennine Fault detecting experiments for sequential circuits , 1964, SWCT 1964.

[46]  Tomás F. Pena,et al.  A parallel 3D semiconductor device simulator for gradual heterojunction bipolar transistors , 2003 .

[47]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[48]  Odile Laurent Using Formal Methods and Testability Concepts in the Avionics Systems Validation and Verification (V&V) Process , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[49]  Mark Harman,et al.  Transition coverage testing for simulink/stateflow models using messy genetic algorithms , 2011, GECCO '11.

[50]  Mark Harman,et al.  Computing Unique Input/Output Sequences Using Genetic Algorithms , 2003, FATES.

[51]  Jesús Carretero,et al.  New techniques for simulating high performance MPI applications on large storage networks , 2008, 2008 IEEE International Conference on Cluster Computing.

[52]  Wolfgang Grieskamp,et al.  Model‐based quality assurance of protocol documentation: tools and methodology , 2011, Softw. Test. Verification Reliab..

[53]  Jesús Carretero,et al.  SIMCAN: A flexible, scalable and expandable simulation platform for modelling and simulating distributed architectures and applications , 2012, Simul. Model. Pract. Theory.

[54]  Rance Cleaveland,et al.  Using formal methods to support testing , 2008 .

[55]  Alexander S. Szalay,et al.  Designing and mining multi-terabyte astronomy archives: the Sloan Digital Sky Survey , 2000, SIGMOD 2000.

[56]  Robert M. Hierons,et al.  Reduced Length Checking Sequences , 2002, IEEE Trans. Computers.

[57]  Justin P. Haldar,et al.  Accelerating advanced mri reconstructions on gpus , 2008, CF '08.