Generating feasible protocol test sequences from EFSM models using Monte Carlo tree search

Abstract Context: Feasible test sequences generation is a key step in protocol conformance testing based on the Extended Finite State Machine (EFSM) model. To guarantee the feasibility of generated test sequences, transition executability analysis (TEA) technique is widely applied in automatic test derivation. However, the TEA method often suffers from the famous state explosion problem, which has become a major obstacle to its efficient application. Objective: In order to mitigate this issue, this paper proposed a novel heuristic TEA method (MTEA) that uses Monte Carlo tree search (MCTS) to guide the TEA tree expansion for efficiently deriving feasible test sequences. Method: The approach first provides a framework to apply the MCTS algorithm based on multiple decision subtrees, in the context of test sequence generation for EFSM-specified systems, to more efficiently expanding the TEA tree with huge state space, and thus alleviating the problem of state explosion. To achieve this, we then design a reward function to calculate the fitness of nodes currently being expanded in the TEA tree and heuristically direct the search towards a near-optimal solution. Next, an adaptive reduction mechanism of search budget is also introduced to accelerate the convergence of the analysis. Finally, a MTEA-based algorithm for automatically generating feasible test sequences is presented under a specific transition coverage criterion. Results: A detailed case study on 6 popular EFSMs was carried out to evaluate the effectiveness and efficiency of our method. Experimental results show that the MTEA significantly outperforms Breadth-First-Search based TEA method (BTEA) and the standard MCTS-based method (SMCTS), regarding time and space performance. Compared with the BTEA, SMCTS and random TEA method (RTEA), the success rate of test generation of MTEA (98.14% on average) is approximately 2, 1.85 and 3 times higher, respectively. For successful test derivation, MTEA only needs to explore on average 9.95% of the nodes and consume on average 61.68% of the runtime of the BTEA method. Conclusion: The experiments illustrate the promise of our approach for alleviating the state explosion problem in test generation for EFSM-specified systems.

[1]  Baowen Xu,et al.  An empirical study on constraint optimization techniques for test generation , 2015, Science China Information Sciences.

[2]  Mark Harman,et al.  Testing conformance of a deterministic implementation against a non-deterministic stream X-machine , 2004, Theor. Comput. Sci..

[3]  Baowen Xu,et al.  EFSM-Based Test Case Generation: Sequence, Data, and Oracle , 2015, Int. J. Softw. Eng. Knowl. Eng..

[4]  Pieter Spronck,et al.  Monte-Carlo Tree Search: A New Framework for Game AI , 2008, AIIDE.

[5]  Markus Neuhäuser,et al.  Wilcoxon Signed Rank Test , 2006 .

[6]  Corina S. Pasareanu,et al.  Statistical symbolic execution with informed sampling , 2014, Software Engineering & Management.

[7]  Gregor von Bochmann,et al.  Some notes on the history of protocol engineering , 2010, Comput. Networks.

[8]  Roberto Baldoni,et al.  A Survey of Symbolic Execution Techniques , 2016, ACM Comput. Surv..

[9]  Zuohua Ding,et al.  A heuristic transition executability analysis method for generating EFSM-specified protocol test sequences , 2016, Inf. Sci..

[10]  Jian Zhang,et al.  Path-oriented test data generation using symbolic execution and constraint solving techniques , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[11]  Mark Harman,et al.  Empirical Study on the Efficiency of Search Based Test Generation for EFSM Models , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[12]  Claudio Bettini,et al.  The Privacy Implications of Cyber Security Systems , 2018, ACM Comput. Surv..

[13]  Robert Feldt,et al.  Heuristic Model Checking using a Monte-Carlo Tree Search Algorithm , 2015, GECCO.

[14]  Alexandre Petrenko,et al.  On Fault Coverage of Tests for Finite State Specifications , 1996, Comput. Networks ISDN Syst..

[15]  Peter Auer,et al.  Finite-time Analysis of the Multiarmed Bandit Problem , 2002, Machine Learning.

[16]  Robert M. Hierons,et al.  On the testability of SDL specifications , 2004, Comput. Networks.

[17]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[18]  Robert M. Hierons,et al.  An integrated search-based approach for automatic testing from extended finite state machine (EFSM) models , 2011, Inf. Softw. Technol..

[19]  Samuel T. Chanson,et al.  A unified approach to protocol test sequence generation , 1993, IEEE INFOCOM '93 The Conference on Computer Communications, Proceedings.

[20]  Byoungju Choi,et al.  Validation of SDL specifications using EFSM-based test generation , 2009, Inf. Softw. Technol..

[21]  Baowen Xu,et al.  A New Approach to Evaluate Path Feasibility and Coverage Ratio of EFSM Based on Multi-objective Optimization , 2012, SEKE.

[22]  Corina S. Pasareanu,et al.  Monte Carlo Tree Search for Finding Costly Paths in Programs , 2018, SEFM.

[23]  Ana R. Cavalli,et al.  FSM-based conformance testing methods: A survey annotated with experimental evaluation , 2010, Inf. Softw. Technol..

[24]  Eliane Martins,et al.  MOST: A Multi-objective Search-Based Testing from EFSM , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[25]  Chung-Ming Huang,et al.  Executable EFSM—based data flow and control flow protocol test sequence generation using reachability analysis , 1999 .

[26]  Mark Harman,et al.  Estimating the feasibility of transition paths in extended finite state machines , 2010, Automated Software Engineering.

[27]  Chung-Ming Huang,et al.  UIOE: protocol test sequence generation method using the transition executability analysis (TEA) , 1998, Comput. Commun..

[28]  Ana R. Cavalli,et al.  New approaches for passive testing using an Extended Finite State Machine specification , 2003, Inf. Softw. Technol..

[29]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[30]  Yvan Labiche,et al.  On the Effect of Counters in Guard Conditions When State-Based Multi-objective Testing , 2015, 2015 IEEE International Conference on Software Quality, Reliability and Security - Companion.

[31]  Guowei Yang,et al.  Chapter Five - Advances in Symbolic Execution , 2019, Adv. Comput..

[32]  Jamal Bentahar,et al.  Chapter Three - Testing the Control-Flow, Data-Flow, and Time Aspects of Communication Systems: A Survey , 2017, Adv. Comput..

[33]  Simon M. Lucas,et al.  A Survey of Monte Carlo Tree Search Methods , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[34]  M. Ümit Uyar,et al.  A method enabling feasible conformance test sequence generation for EFSM models , 2004, IEEE Transactions on Computers.

[35]  Alexandre Petrenko,et al.  Confirming configurations in EFSM testing , 2004, IEEE Transactions on Software Engineering.

[36]  Robert M. Hierons,et al.  Expanding an extended finite state machine to aid testability , 2002, Proceedings 26th Annual International Computer Software and Applications.

[37]  Valdivino Alexandre de Santiago Júnior,et al.  A systematic mapping addressing Hyper-Heuristics within Search-based Software Testing , 2019, Inf. Softw. Technol..

[38]  Rachida Dssouli,et al.  Automatic executable test case generation for extended finite state machine protocols , 1997 .

[39]  Ana R. Cavalli,et al.  An EFSM-Based Intrusion Detection System for Ad Hoc Networks , 2005, ATVA.

[40]  Bo Yang,et al.  A test sequence selection method for protocol testing , 1991, IEEE Trans. Commun..

[41]  Dong-Jin Lim,et al.  Automatic feasible transition path generation from UML state chart diagrams using grouping genetic algorithms , 2018, Inf. Softw. Technol..