Evaluating model testing and model checking for finding requirements violations in Simulink models

Matlab/Simulink is a development and simulation language that is widely used by the Cyber-Physical System (CPS) industry to model dynamical systems. There are two mainstream approaches to verify CPS Simulink models: model testing that attempts to identify failures in models by executing them for a number of sampled test inputs, and model checking that attempts to exhaustively check the correctness of models against some given formal properties. In this paper, we present an industrial Simulink model benchmark, provide a categorization of different model types in the benchmark, describe the recurring logical patterns in the model requirements, and discuss the results of applying model checking and model testing approaches to identify requirements violations in the benchmarked models. Based on the results, we discuss the strengths and weaknesses of model testing and model checking. Our results further suggest that model checking and model testing are complementary and by combining them, we can significantly enhance the capabilities of each of these approaches individually. We conclude by providing guidelines as to how the two approaches can be best applied together.

[1]  Edmund M. Clarke,et al.  Statistical Model Checking for Cyber-Physical Systems , 2011, ATVA.

[2]  Lionel C. Briand,et al.  Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior , 2019, IEEE Transactions on Software Engineering.

[3]  Lionel C. Briand,et al.  Testing advanced driver assistance systems using multi-objective search and neural networks , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[4]  Lionel C. Briand,et al.  Automated testing of hybrid Simulink/Stateflow controllers: industrial case studies , 2017, ESEC/SIGSOFT FSE.

[5]  Mehrdad Sabetzadeh,et al.  Testing the Untestable - Model Testing of Complex Software-Intensive Systems , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[6]  Kerianne H. Gross,et al.  Incremental Formal Methods Based Design Approach Demonstrated on a Coupled Tanks Control System , 2016, 2016 IEEE 17th International Symposium on High Assurance Systems Engineering (HASE).

[7]  Håkan L. S. Younes,et al.  Statistical probabilistic model checking with a focus on time-bounded properties , 2006, Inf. Comput..

[8]  Pravin Varaiya,et al.  What's decidable about hybrid automata? , 1995, STOC '95.

[9]  Lionel C. Briand,et al.  MiL testing of highly configurable continuous controllers: scalable search using surrogate models , 2014, ASE.

[10]  Sriram Sankaranarayanan,et al.  Probabilistic Temporal Logic Falsification of Cyber-Physical Systems , 2013, TECS.

[11]  Pieter J. Mosterman,et al.  Model-Based Testing for Embedded Systems , 2011, Computational Analysis, Synthesis, & Design Dynamic Systems.

[12]  Lionel C. Briand,et al.  Automated Test Suite Generation for Time-Continuous Simulink Models , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[13]  Axel Legay,et al.  Statistical Model Checking: An Overview , 2010, RV.

[14]  Christine Julien,et al.  Perceptions on the State of the Art in Verification and Validation in Cyber-Physical Systems , 2017, IEEE Systems Journal.

[15]  Rajeev Alur,et al.  Formal verification of hybrid systems , 2011, 2011 Proceedings of the Ninth ACM International Conference on Embedded Software (EMSOFT).

[16]  Rajeev Alur,et al.  Principles of Cyber-Physical Systems , 2015 .

[17]  Thomas A. Henzinger,et al.  The Algorithmic Analysis of Hybrid Systems , 1995, Theor. Comput. Sci..

[18]  Kim G. Larsen,et al.  A modal process logic , 1988, [1988] Proceedings. Third Annual Information Symposium on Logic in Computer Science.

[19]  Daniel Kroening,et al.  Software Verification Using k-Induction , 2011, SAS.

[20]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[21]  Norman S. Nise,et al.  Control Systems Engineering , 1991 .

[22]  Rance Cleaveland,et al.  An Instrumentation-Based Approach to Controller Model Validation , 2006, ASWSD.

[23]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[24]  Armin Biere,et al.  Bounded Model Checking Using Satisfiability Solving , 2001, Formal Methods Syst. Des..

[25]  Amir Pnueli,et al.  The temporal logic of programs , 1977, 18th Annual Symposium on Foundations of Computer Science (sfcs 1977).

[26]  Phil McMinn,et al.  Search-based software test data generation: a survey: Research Articles , 2004 .

[27]  Lionel C. Briand,et al.  Search-based automated testing of continuous controllers: Framework, tool support, and case studies , 2015, Inf. Softw. Technol..

[28]  Dejan Nickovic,et al.  Monitoring Temporal Properties of Continuous Signals , 2004, FORMATS/FTRTFT.

[29]  Edmund M. Clarke,et al.  Bayesian statistical model checking with application to Stateflow/Simulink verification , 2013, Formal Methods Syst. Des..

[30]  Lionel C. Briand,et al.  Generating automated and online test oracles for Simulink models with continuous and uncertain behaviors , 2019, ESEC/SIGSOFT FSE.

[31]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[32]  Luciano Baresi,et al.  Test oracles for simulink-like models , 2016, Automated Software Engineering.

[33]  George J. Pappas,et al.  Robustness of temporal logic specifications for continuous-time signals , 2009, Theor. Comput. Sci..

[34]  Lubos Brim,et al.  Executing Model Checking Counterexamples in Simulink , 2012, 2012 Sixth International Symposium on Theoretical Aspects of Software Engineering.