Model-based testing for software safety: a systematic mapping study

Testing safety-critical systems is crucial since a failure or malfunction may result in death or serious injuries to people, equipment, or environment. An important challenge in testing is the derivation of test cases that can identify the potential faults. Model-based testing adopts models of a system under test and/or its environment to derive test artifacts. This paper aims to provide a systematic mapping study to identify, analyze, and describe the state-of-the-art advances in model-based testing for software safety. The systematic mapping study is conducted as a multi-phase study selection process using the published literature in major software engineering journals and conference proceedings. We reviewed 751 papers and 36 of them have been selected as primary studies to answer our research questions. Based on the analysis of the data extraction process, we discuss the primary trends and approaches and present the identified obstacles. This study shows that model-based testing can provide important benefits for software safety testing. Several solution directions have been identified, but further research is critical for reliable model-based testing approach for safety.

[1]  Alan Hartman,et al.  Choosing a Test Modeling Language: A Survey , 2006, Haifa Verification Conference.

[2]  Heiko Stallbaum,et al.  Toward DO-178B-compliant Test Models , 2010, 2010 Workshop on Model-Driven Engineering, Verification, and Validation.

[3]  Jeffrey D. Ullman,et al.  Introduction to automata theory, languages, and computation, 2nd edition , 2001, SIGA.

[4]  Stacy J. Prowell,et al.  JUMBL: a tool for model-based statistical testing , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[5]  G. Guyatt,et al.  Grading quality of evidence and strength of recommendations , 2004, BMJ : British Medical Journal.

[6]  M. W. Shields.  An Introduction to Automata Theory , 1988 .

[7]  Bernhard K. Aichernig,et al.  UML in action: a two-layered interpretation for testing , 2011, SOEN.

[8]  Takashi Kitamura,et al.  Formal Model-Based Test for AUTOSAR Multicore RTOS , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[9]  Paul Pettersson,et al.  Cover - A Real-Time Test Case Generation Tool , 2007 .

[10]  Sébastien Gérard,et al.  Towards a Framework for Explicit Platform-Based Transformations , 2008, 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC).

[11]  Elaine J. Weyuker,et al.  Automated test generation using model checking: an industrial evaluation , 2014, International Journal on Software Tools for Technology Transfer.

[12]  Rajesh Subramanyan,et al.  A survey on model-based testing approaches: a systematic review , 2007, WEASELTech '07.

[13]  Dejan Nickovic,et al.  Scalable Incremental Test-case Generation from Large Behavior Models , 2015, TAP@STAF.

[14]  Paolo Arcaini,et al.  Rigorous development process of a safety-critical system: from ASM models to Java code , 2017, International Journal on Software Tools for Technology Transfer.

[15]  Robert Eschbach,et al.  Risk-Based Testing of Safety-Critical Embedded Systems Driven by Fault Tree Analysis , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[16]  Alessio Ferrari,et al.  Model Based Testing and Abstract Interpretation in the Railway Signaling Context , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[17]  Florian Lorber,et al.  Debugging with Timed Automata Mutations , 2014, SAFECOMP.

[18]  G. Guyatt,et al.  Grading quality of evidence and strength of recommendations , 2004, British medical journal.

[19]  Bernhard K. Aichernig,et al.  Mapping UML to Labeled Transition Systems for Test-Case Generation - A Translation via Object-Oriented Action Systems , 2009, FMCO.

[20]  Ratnesh Kumar,et al.  Stateflow to Extended Finite Automata Translation , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops.

[21]  Andy Schürr,et al.  Model-based coverage-driven test suite generation for software product lines , 2011, MODELS'11.

[22]  Karsten Berns,et al.  A Systematic Testing Approach for Autonomous Mobile Robots Using Domain-Specific Languages , 2010, KI.

[23]  Stefano Marrone,et al.  Test Specification Patterns for Automatic Generation of Test Sequences , 2014, FMICS.

[24]  Jure Leskovec,et al.  Mining Social-Network Graphs , 2014 .

[25]  James Bret Michael,et al.  Environment behavior models for automation of testing and assessment of system safety , 2006, Inf. Softw. Technol..

[26]  Alain Abran,et al.  The Guide to the Software Engineering Body of Knowledge , 1999, IEEE Softw..

[27]  Zoltán Szatmári,et al.  A Concept for Testing Robustness and Safety of the Context-Aware Behaviour of Autonomous Systems , 2012, KES-AMSTA.

[28]  Daniel Kroening,et al.  Generating test case chains for reactive systems , 2014, International Journal on Software Tools for Technology Transfer.

[29]  Yuen Man Hon,et al.  Generating Scenarios by Multi-Object Checking , 2007, MBT.

[30]  Michael J. Butler,et al.  A Systematic Approach to Requirements Driven Test Generation for Safety Critical Systems , 2014, IMBSA.

[31]  Gang Yu,et al.  An Approach for Automated Safety Testing of Safety-Critical Software System Based on Safety Requirements , 2009, 2009 International Forum on Information Technology and Applications.

[32]  Kai Petersen,et al.  Benefits and limitations of automated software testing: Systematic literature review and practitioner survey , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[33]  Malte Lochau,et al.  Feature Interaction Aware Test Case Generation for Embedded Control Systems , 2010, Electron. Notes Theor. Comput. Sci..

[34]  Wolfgang Herzner,et al.  Towards generation of efficient test cases from UML/OCL models for complex safety-critical systems , 2010, Elektrotech. Informationstechnik.

[35]  Gordon Fraser,et al.  Testing with model checkers: a survey , 2009 .

[36]  Kim G. Larsen,et al.  Formal Analysis and Testing of Real-Time Automotive Systems Using UPPAAL Tools , 2015, FMICS.

[37]  Stefano Marrone,et al.  Towards Model-Driven V&V assessment of railway control systems , 2014, International Journal on Software Tools for Technology Transfer.

[38]  Mathieu Acher,et al.  Deriving Usage Model Variants for Model-Based Testing: An Industrial Case Study , 2014, 2014 19th International Conference on Engineering of Complex Computer Systems.

[39]  Gang Yu,et al.  Model-Based Safety Test Automation of Safety-Critical Software , 2010, 2010 International Conference on Computational Intelligence and Software Engineering.

[40]  Sandeep K. Shukla,et al.  Model-driven test generation for system level validation , 2007, 2007 IEEE International High Level Design Validation and Test Workshop.

[41]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[42]  S. Ramesh,et al.  Automatic test case generation from Simulink/Stateflow models using model checking , 2014, Softw. Test. Verification Reliab..

[43]  Bruno Legeard,et al.  A taxonomy of model-based testing , 2006 .

[44]  Tao Tang,et al.  Model-based test cases generation for Onboard system , 2013, 2013 IEEE Eleventh International Symposium on Autonomous Decentralized Systems (ISADS).

[45]  Kurt Jensen,et al.  Coloured Petri Nets , 1997, Monographs in Theoretical Computer Science An EATCS Series.

[46]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[47]  Anneliese Amschler Andrews,et al.  Fail-safe testing of safety-critical systems: a case study and efficiency analysis , 2015, Software Quality Journal.

[48]  R. Dupuis,et al.  Guide to the software engineering body of knowledge stakeholder issues and intended usages , 1999, Proceedings 4th IEEE International Software Engineering Standards Symposium and Forum (ISESS'99). 'Best Software Practices for the Internet Age'.

[49]  Daniel Sundmark,et al.  Model-Based Test Suite Generation for Function Block Diagrams Using the UPPAAL Model Checker , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[50]  Robert Eschbach,et al.  A Systematic Approach to Construct Compositional Behaviour Models for Network-structured Safety-critical Systems , 2010, Electron. Notes Theor. Comput. Sci..

[51]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[52]  Yunja Choi,et al.  Constraint-based test generation for automotive operating systems , 2017, Software & Systems Modeling.

[53]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[54]  Angelo Gargantini.  Using Model Checking to Generate Fault Detecting Tests , 2007, TAP.

[55]  Chin-Feng Fan,et al.  Systematic scenario test case generation for nuclear safety systems , 2013, Inf. Softw. Technol..

[56]  Muhammad Ali Babar,et al.  Identifying relevant studies in software engineering , 2011, Inf. Softw. Technol..

[57]  Paul Pettersson,et al.  A Global Algorithm for Model-Based Test Suite Generation , 2007, Electron. Notes Theor. Comput. Sci..

[58]  Raimund Kirner,et al.  Development of a Framework for Automated Systematic Testing of Safety-Critical Embedded Systems , 2006, 2006 International Workshop on Intelligent Solutions in Embedded Systems.

[59]  Alain Abran,et al.  Guide to the Software Engineering Body of Knowledge : 2004 Version , 2005 .

[60]  Rolf Drechsler,et al.  Formal Modeling and Verification of Cyber-Physical Systems , 2015, Springer Fachmedien Wiesbaden.

[61]  Rui Wang,et al.  Automated Test Approach Based on All Paths Covered Optimal Algorithm and Sequence Priority Selected Algorithm , 2014, IEEE Transactions on Intelligent Transportation Systems.