A Systematic and Semi-Automatic Safety-Based Test Case Generation Approach Based on Systems-Theoretic Process Analysis

Software safety is a crucial aspect during the development of modern safety-critical systems. Software is becoming responsible for most of the critical functions of systems. Therefore, the software components in the systems need to be tested extensively against their safety requirements to ensure a high level of system safety. However, performing testing exhaustively to test all software behaviours is impossible. Numerous testing approaches exist. However, they do not directly concern the information derived during the safety analysis. STPA (Systems-Theoretic Process Analysis) is a unique safety analysis approach based on system and control theory, and was developed to identify unsafe scenarios of a complex system including software. In this paper, we present a systematic and semi-automatic testing approach based on STPA to generate test cases from the STPA safety analysis results to help software and safety engineers to recognize and reduce the associated software risks. We also provide an open-source safety-based testing tool called STPA TCGenerator to support the proposed approach. We illustrate the proposed approach with a prototype of a software of the Adaptive Cruise Control System (ACC) with a stop-and-go function with a Lego-Mindstorms EV3 robot.

[1]  Andrew P. Martin,et al.  A Set-Theoretic Model for Real-Time Specification and Reasoning , 1998, MPC.

[2]  Jeffrey S. Lavell,et al.  Report on the Loss of the Mars Polar Lander and Deep Space 2 Missions , 2000 .

[3]  Nancy G. Leveson,et al.  Engineering a Safer World: Systems Thinking Applied to Safety , 2012 .

[4]  Stefan Wagner,et al.  Integrated Safety Analysis Using Systems-Theoretic Process Analysis and Software Model Checking , 2015, SAFECOMP.

[5]  Thomas Bauer,et al.  Risk-based Statistical Testing: A Refinement- based Approach to the Reliability Analysis of Safety-Critical Systems , 2009 .

[6]  Robert Eschbach,et al.  Risk-Based Testing of Safety-Critical Embedded Systems Driven by Fault Tree Analysis , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[7]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures , 2005 .

[8]  Yu Lei,et al.  Introduction to Combinatorial Testing , 2013 .

[9]  Paul J.Th. Venhovens,et al.  Stop and Go Cruise Control , 2000 .

[10]  John M. Rushby,et al.  An operational semantics for Stateflow , 2004, International Journal on Software Tools for Technology Transfer.

[11]  Evaluating the Safety of Digital Instrumentation and Control Systems in Nuclear Power Plants , 2012 .

[12]  Doo-Hwan Bae,et al.  A test sequence selection method for statecharts , 2000, Softw. Test. Verification Reliab..

[13]  Fausto Giunchiglia,et al.  NUSMV: a new symbolic model checker , 2000, International Journal on Software Tools for Technology Transfer.

[14]  John P. Thomas,et al.  Extending and automating a systems-theoretic hazard analysis for requirements generation and analysis , 2013 .

[15]  Stefan Wagner,et al.  A comprehensive safety engineering approach for software-intensive systems based on STPA , 2015, ArXiv.

[16]  Gabor Karsai,et al.  Model Based Analysis and Test Generation for Flight Software , 2009, 2009 Third IEEE International Conference on Space Mission Challenges for Information Technology.

[17]  Hasan Ural.  Test sequence selection based on static data flow analysis , 1987, Comput. Commun..

[18]  Ratnesh Kumar,et al.  Model-based automatic test generation for Simulink/Stateflow using extended finite automaton , 2012, 2012 IEEE International Conference on Automation Science and Engineering (CASE).

[19]  Andreas Hoffmann,et al.  Model-Based Testing , 2012, IEEE Software.

[20]  Lakhdar Sais,et al.  ManySAT: a Parallel SAT Solver , 2009, J. Satisf. Boolean Model. Comput..

[21]  Robyn R. Lutz,et al.  Engineering for Safety : A Roadmap , 2001 .

[22]  Andreas Windisch,et al.  Search-based testing of complex simulink models containing stateflow diagrams , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[23]  Leon J. Osterweil,et al.  Data Flow Analysis in Software Reliability , 1976, CSUR.

[24]  Felix Redmill.  Exploring risk-based testing and its implications: Research Articles , 2004 .

[25]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[26]  K. Butts,et al.  Symbolic verification of executable control specifications , 1999, Proceedings of the 1999 IEEE International Symposium on Computer Aided Control System Design (Cat. No.99TH8404).

[27]  Kenneth L. McMillan,et al.  Symbolic model checking , 1992 .

[28]  Rachida Dssouli,et al.  Automatic executable test case generation for extended finite state machine protocols , 1997 .

[29]  Jin Song Dong,et al.  Applying Timed Interval Calculus to Simulink Diagrams , 2006, ICFEM.

[30]  Sudeepa Roy,et al.  Tool for Translating Simulink Models into Input Language of a Model Checker , 2006, ICFEM.

[31]  George H. Mealy,et al.  A method for synthesizing sequential circuits , 1955 .

[32]  Doo-Hwan Bae,et al.  Test cases generation from UML state diagrams , 1999, IEE Proc. Softw..

[33]  A. Pretschner.  Classical search strategies for test case generation with Constraint Logic Programming , 2001 .

[34]  John A. Clark,et al.  A search-based framework for automatic testing of MATLAB/Simulink models , 2008, J. Syst. Softw..

[35]  Kwang-Ting Cheng,et al.  Automatic Functional Test Generation Using The Extended Finite State Machine Model , 1993, 30th ACM/IEEE Design Automation Conference.

[36]  R. C. Bromley,et al.  Failure modes, effects and criticality analysis (FMECA) , 1994 .

[37]  Jun Sun,et al.  Integrating Specification and Programs for System Modeling and Verification , 2009, 2009 Third IEEE International Symposium on Theoretical Aspects of Software Engineering.

[38]  Grégoire Hamon,et al.  A denotational semantics for stateflow , 2005, EMSOFT.

[39]  Sérgio Vale Aguiar Campos,et al.  Symbolic Model Checking , 1993, CAV.

[40]  Nancy G. Leveson.  Completeness in formal specification language design for process-control systems , 2000, FMSP '00.

[41]  S. Seshu,et al.  Introduction to the theory of finite-state machines , 1963 .

[42]  Stacy J. Prowell,et al.  JUMBL: a tool for model-based statistical testing , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[43]  David Harel,et al.  Statecharts: A Visual Formalism for Complex Systems , 1987, Sci. Comput. Program..

[44]  Amnon Naamad,et al.  The STATEMATE semantics of statecharts , 1996, TSEM.

[45]  Christos Sofronis,et al.  Parallel NuSMV: A NuSMV Extension for the Verification of Complex Embedded Systems , 2012, SAFECOMP Workshops.

[46]  Asim Abdulkhaleq,et al.  XSTAMPP: An eXtensible STAMP platform as tool support for safety engineering , 2015 .

[47]  Jun Sun,et al.  Formal modeling and validation of Stateflow diagrams , 2012, International Journal on Software Tools for Technology Transfer.

[48]  Richard Castanet,et al.  Generate Certified Test Cases by Combining Theorem Proving and Reachability Analysis , 2002, TestCom.

[49]  Rajeev Alur,et al.  Timed Automata , 1999, CAV.

[50]  Jun Sun,et al.  PAT: Towards Flexible Verification under Fairness , 2009, CAV.

[51]  J. Thomas.  Performing Hazard Analysis on Complex, Software- and Human-Intensive Systems , 2011 .

[52]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems: Advanced Lectures (Lecture Notes in Computer Science) , 2005 .

[53]  Shaoying Liu,et al.  Generating test data from state‐based specifications , 2003, Softw. Test. Verification Reliab..

[54]  Yu Jiang,et al.  Verifying Simulink Stateflow model: Timed automata approach , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[55]  Fausto Giunchiglia,et al.  NUSMV: A New Symbolic Model Verifier , 1999, CAV.