Efficient coverage of parallel and hierarchical stateflow models for test case generation

This paper is concerned with test case generation from Simulink/Stateflow (SL/SF) models with a focus on coverage of SF model elements. Coverage of the SF component in a model is a difficult task because of two primary reasons: (i) the SF component itself may lie deep in the SL/SF model in which case, inputs have to pass through a complex chain of SL blocks to reach the SF block and (ii) nonlinear constraints in the model are difficult to solve using constraint solvers. Hierarchy and parallelism in the SF model add further complexity to the problem. The existing approaches flatten such SF elements, and generate test cases from the flattened finite state machines. Handling of issues (i) and (ii) has already been discussed in earlier research. In this paper, we present a method of covering SF components, which does not require to flatten any hierarchy or parallelism in the components. This not only makes the test case generation problem efficient but also addresses the problem of scalability. We have implemented this method and performed a number of medium‐sized case studies. The results show improved performance over the results obtained by some commercial tools. Copyright © 2011 John Wiley & Sons, Ltd.

[1]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[2]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[3]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[4]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[5]  Mohammad Alshraideh,et al.  Search‐based software test data generation for string data using program‐specific search operators , 2006, Softw. Test. Verification Reliab..

[6]  Per Bjesse,et al.  Using counter example guided abstraction refinement to find complex bugs , 2004, Proceedings Design, Automation and Test in Europe Conference and Exhibition.

[7]  Mark Harman,et al.  Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet? , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[8]  Stefan Wappler,et al.  Evolutionary functional testing of continuous control systems , 2009, GECCO.

[9]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[10]  Rance Cleaveland,et al.  An Instrumentation-Based Approach to Controller Model Validation , 2006, ASWSD.

[11]  Rajeev Alur,et al.  Symbolic analysis for improving simulation coverage of Simulink/Stateflow models , 2008, EMSOFT '08.

[12]  Shaoying Liu,et al.  Generating test data from state‐based specifications , 2003, Softw. Test. Verification Reliab..

[13]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[14]  Stavros Tripakis,et al.  Defining and translating a "safe" subset of simulink/stateflow into lustre , 2004, EMSOFT '04.

[15]  Helmut Veith,et al.  Counterexample-guided abstraction refinement for symbolic model checking , 2003, JACM.

[16]  Insup Lee,et al.  Robust Test Generation and Coverage for Hybrid Systems , 2007, HSCC.

[17]  Etsi Tc-Mts.  Methods for Testing and Specification (MTS); Protocol and profile conformance testing specifications; Standardization methodology , 1995 .

[18]  Mirko Conrad,et al.  Systematic Model-Based Testing of Embedded Automotive Software , 2004, MBT.

[19]  S. Ramesh,et al.  Randomized directed testing (REDIRECT) for Simulink/Stateflow models , 2008, EMSOFT '08.

[20]  Bogdan Korel,et al.  Slicing of state-based models , 2003, International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings..

[21]  Leonardo de Moura,et al.  Automated Test Generation with SAL , 2005 .

[22]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[23]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[24]  Andreas Rau,et al.  On Modelling Guidelines: Flowchart Patterns for STATEFLOW , 2001, Softwaretechnik-Trends.

[25]  Nicolas Halbwachs,et al.  Automatic testing of reactive systems , 1998, Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No.98CB36279).

[26]  Andreas Windisch,et al.  Search-based testing of complex simulink models containing stateflow diagrams , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[27]  John A. Clark,et al.  A search-based framework for automatic testing of MATLAB/Simulink models , 2008, J. Syst. Softw..

[28]  Experience report: the reactis validation tool , 2007, ICFP '07.

[29]  Nicolas Halbwachs,et al.  Validation of Synchronous Reactive Systems: From Formal Verification to Automatic Testing , 1999, ASIAN.

[30]  Rance Cleaveland,et al.  Automated validation of software models , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[31]  S. Ramesh,et al.  AutoMOTGen: Automatic Model Oriented Test Generator for Embedded Control Systems , 2008, CAV.

[32]  André Baresel,et al.  Structural and Functional Sequence Test of Dynamic and State-Based Software with Evolutionary Algorithms , 2003, GECCO.

[33]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[34]  Oliver Bühler,et al.  Evolutionary functional testing , 2008, Comput. Oper. Res..

[35]  John M. Rushby,et al.  An operational semantics for Stateflow , 2004, International Journal on Software Tools for Technology Transfer.