An integrated search-based approach for automatic testing from extended finite state machine (EFSM) models

Context: The extended finite state machine (EFSM) is a modelling approach that has been used to represent a wide range of systems. When testing from an EFSM, it is normal to use a test criterion such as transition coverage. Such test criteria are often expressed in terms of transition paths (TPs) through an EFSM. Despite the popularity of EFSMs, testing from an EFSM is difficult for two main reasons: path feasibility and path input sequence generation. The path feasibility problem concerns generating paths that are feasible whereas the path input sequence generation problem is to find an input sequence that can traverse a feasible path. Objective: While search-based approaches have been used in test automation, there has been relatively little work that uses them when testing from an EFSM. In this paper, we propose an integrated search-based approach to automate testing from an EFSM. Method: The approach has two phases, the aim of the first phase being to produce a feasible TP (FTP) while the second phase searches for an input sequence to trigger this TP. The first phase uses a Genetic Algorithm whose fitness function is a TP feasibility metric based on dataflow dependence. The second phase uses a Genetic Algorithm whose fitness function is based on a combination of a branch distance function and approach level. Results: Experimental results using five EFSMs found the first phase to be effective in generating FTPs with a success rate of approximately 96.6%. Furthermore, the proposed input sequence generator could trigger all the generated feasible TPs (success rate=100%). Conclusion: The results derived from the experiment demonstrate that the proposed approach is effective in automating testing from an EFSM.

[1]  Gregor von Bochmann,et al.  Specifications of a Simplified Transport Protocol Using Different Formal Description Techniques , 1990, Comput. Networks ISDN Syst..

[2]  Kwang-Ting Cheng,et al.  Automatic generation of functional vectors using the extended finite state machine model , 1996, TODE.

[3]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[4]  Bo Yang,et al.  A test sequence selection method for protocol testing , 1991, IEEE Trans. Commun..

[5]  Mark Weiser,et al.  Program Slicing , 1981, IEEE Transactions on Software Engineering.

[6]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[7]  M. Ümit Uyar,et al.  A method enabling feasible conformance test sequence generation for EFSM models , 2004, IEEE Transactions on Computers.

[8]  Krishnaiyan Thulasiraman,et al.  Context independent unique state identification sequences for testing communication protocols modelled as extended finite state machines , 2003, Comput. Commun..

[9]  Jing-Yang Jou,et al.  Stimulus generation for interface protocol verification using the nondeterministic extended finite state machine model , 2005, Tenth IEEE International High-Level Design Validation and Test Workshop, 2005..

[10]  Bogdan Korel,et al.  Requirement-based automated black-box test generation , 2001, 25th Annual International Computer Software and Applications Conference. COMPSAC 2001.

[11]  Bogdan Korel,et al.  Model based regression test reduction using dependence analysis , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[12]  Jian Zhang,et al.  Path-oriented test data generation using symbolic execution and constraint solving techniques , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[13]  John A. Clark,et al.  Formulating software engineering as a search problem , 2003, IEE Proc. Softw..

[14]  Sadiq M. Sait,et al.  Iterative computer algorithms with applications in engineering - solving combinatorial optimization problems , 2000 .

[15]  Samuel T. Chanson,et al.  Automatic protocol test suite derivation , 1994, Proceedings of INFOCOM '94 Conference on Computer Communications.

[16]  Rance Cleaveland,et al.  Using formal specifications to support testing , 2009, CSUR.

[17]  Robert M. Hierons,et al.  On the testability of SDL specifications , 2004, Comput. Networks.

[18]  Samuel T. Chanson,et al.  A unified approach to protocol test sequence generation , 1993, IEEE INFOCOM '93 The Conference on Computer Communications, Proceedings.

[19]  Florentin Ipate,et al.  Functional Search-based Testing from State Machines , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[20]  Kalyanmoy Deb,et al.  Genetic Algorithms, Noise, and the Sizing of Populations , 1992, Complex Syst..

[21]  C. Williams,et al.  On Generating EFSM Models from Use Cases , 2007, Sixth International Workshop on Scenarios and State Machines (SCESM'07: ICSE Workshops 2007).

[22]  Krishan K. Sabnani,et al.  Formal methods for generating protocol conformance test sequences , 1990, Proc. IEEE.

[23]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[24]  Stanislaw Budkowski,et al.  An Introduction to Estelle: A Specification Language for Distributed Systems , 1987, Comput. Networks.

[25]  Abdeslam En-Nouaary,et al.  Test development for communication protocols: towards automation , 1999, Comput. Networks.

[26]  Mark Harman,et al.  Estimating the feasibility of transition paths in extended finite state machines , 2010, Automated Software Engineering.

[27]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[28]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[29]  Robert M. Hierons,et al.  Testing a system specified using Statecharts and Z , 2001, Inf. Softw. Technol..

[30]  Kuo-Chung Tai A program complexity metric based on data flow information in control graphs , 1984, ICSE '84.

[31]  M. U. Uyar,et al.  Conformance testing: towards refining VHDL specifications , 1999, MILCOM 1999. IEEE Military Communications. Conference Proceedings (Cat. No.99CH36341).

[32]  Felix Naumann,et al.  Data fusion , 2009, CSUR.

[33]  Robert M. Hierons,et al.  Generating Feasible Transition Paths for Testing from an Extended Finite State Machine (EFSM) , 2009, 2009 International Conference on Software Testing Verification and Validation.

[34]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[35]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[36]  John A. Clark,et al.  The Way Forward for Unifying Dynamic Test Case Generation: The Optimisation-based Approach , 1998 .

[37]  Rachida Dssouli,et al.  Automatic Test Generation for EFSM-based Systems , 1996 .

[38]  Hasan Ural,et al.  Test generation based on control and data dependencies within system specifications in SDL , 2000, Comput. Commun..

[39]  Mark Harman,et al.  Generating feasible input sequences for extended finite state machines (EFSMs) using genetic algorithms , 2005, GECCO '05.

[40]  Leonardo Mariani,et al.  Automatic generation of software behavioral models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[41]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[42]  James C. King,et al.  Applications of Symbolic Execution to Program Testing , 1978, Computer.

[43]  Mark Harman,et al.  Testing conformance of a deterministic implementation against a non-deterministic stream X-machine , 2004, Theor. Comput. Sci..

[44]  Ming T. Liu,et al.  Test path selection based on effective domains , 1994, Proceedings of ICNP - 1994 International Conference on Network Protocols.

[45]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[46]  Stephen Swift,et al.  A Search-Based Approach for Automatic Test Generation from Extended Finite State Machine (EFSM) , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[47]  Gregor von Bochmann,et al.  A Test Design Methodology for Protocol Testing , 1987, IEEE Transactions on Software Engineering.

[48]  Alexandre Petrenko,et al.  On Fault Coverage of Tests for Finite State Specifications , 1996, Comput. Networks ISDN Syst..

[49]  John A. Clark,et al.  Metrics are fitness functions too , 2004 .

[50]  Sungwon Kang,et al.  Generating Test Cases for Web Services Using Extended Finite State Machine , 2006, TestCom.

[51]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).

[52]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[53]  John A. Clark,et al.  Automated program flaw finding using simulated annealing , 1998, ISSTA '98.

[54]  Jian Zhang,et al.  Constraint Solving and Symbolic Execution , 2005, VSTTE.

[55]  Raimund Kirner,et al.  Measurement-Based Timing Analysis , 2008, ISoLA.

[56]  Jonas Mellin,et al.  Test Case Generation for Mutation-based Testing of Timeliness , 2006, Electron. Notes Theor. Comput. Sci..

[57]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[58]  Lionel C. Briand,et al.  Improving the coverage criteria of UML state machines using data flow analysis , 2010 .

[59]  Byoungju Choi,et al.  Validation of SDL specifications using EFSM-based test generation , 2009, Inf. Softw. Technol..

[60]  Lalit M. Patnaik,et al.  Genetic algorithms: a survey , 1994, Computer.

[61]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[62]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[63]  D. Carroll Chemical laser modeling with genetic algorithms , 1996 .

[64]  Alexandre Petrenko,et al.  Confirming configurations in EFSM testing , 2004, IEEE Transactions on Software Engineering.