Adaptive Testing for Specification Coverage

Ensuring correctness of cyber-physical systems (CPS) is an extremely challenging task that is in practice often addressed with simulation based testing. Formal specification languages, such as Signal Temporal Logic (STL), are used to mathematically express CPS requirements and thus render the simulation activity more systematic and principled. We propose a novel method for adaptive generation of tests with specification coverage for STL. To achieve this goal, we devise cooperative reachability games that we combine with numerical optimization to create tests that explore the system in a way that exercise various parts of the specification. To the best of our knowledge our approach is the first adaptive testing approach that can be applied directly to MATLAB\texttrademark\; Simulink/Stateflow models. We implemented our approach in a prototype tool and evaluated it on several illustrating examples and a case study from the avionics domain, demonstrating the effectiveness of adaptive testing to (1) incrementally build a test case that reaches a test objective, (2) generate a test suite that increases the specification coverage, and (3) infer what part of the specification is actually implemented.

[1]  Dejan Nickovic,et al.  Interface-aware signal temporal logic , 2019, HSCC.

[2]  Dejan Nickovic,et al.  Monitoring properties of analog and mixed-signal circuits , 2012, International Journal on Software Tools for Technology Transfer.

[3]  Sriram Sankaranarayanan,et al.  Probabilistic Temporal Logic Falsification of Cyber-Physical Systems , 2013, TECS.

[4]  Muhammad Shafique,et al.  A Roadmap Toward the Resilient Internet of Things for Cyber-Physical Systems , 2018, IEEE Access.

[5]  Sriram Sankaranarayanan,et al.  Requirements driven falsification with coverage metrics , 2015, 2015 International Conference on Embedded Software (EMSOFT).

[6]  Sriram Sankaranarayanan,et al.  Monte-carlo techniques for falsification of temporal properties of non-linear hybrid systems , 2010, HSCC '10.

[7]  George J. Pappas,et al.  Robustness of temporal logic specifications for continuous-time signals , 2009, Theor. Comput. Sci..

[8]  Dejan Nickovic,et al.  Specification-Based Monitoring of Cyber-Physical Systems: A Survey on Theory, Tools and Applications , 2018, Lectures on Runtime Verification.

[9]  Calin Belta,et al.  A Formal Methods Approach to Pattern Recognition and Synthesis in Reaction Diffusion Networks , 2018, IEEE Transactions on Control of Network Systems.

[10]  Yashwanth Singh Rahul Annapureddy,et al.  Ant colonies for Temporal Logic falsification of hybrid systems , 2010, IECON 2010 - 36th Annual Conference on IEEE Industrial Electronics Society.

[11]  Thierry Jéron,et al.  Control strategies for off-line testing of timed systems , 2018, Formal Methods in System Design.

[12]  Ichiro Hasuo,et al.  Time Robustness in MTL and Expressivity in Hybrid System Falsification , 2015, CAV.

[13]  Sriram Sankaranarayanan,et al.  S-TaLiRo: A Tool for Temporal Logic Falsification for Hybrid Systems , 2011, TACAS.

[14]  Roderick Bloem,et al.  Synthesizing adaptive test strategies from temporal logic specifications , 2016, 2016 Formal Methods in Computer-Aided Design (FMCAD).

[15]  Andreas Fellner,et al.  Model-based, mutation-driven test case generation via heuristic-guided branching search , 2017, MEMOCODE.

[16]  Luca Bortolussi,et al.  Bayesian Statistical Parameter Synthesis for Linear Temporal Properties of Stochastic Models , 2018, TACAS.

[17]  Roderick Bloem,et al.  Case Study: Automatic Test Case Generation for a Secure Cache Implementation , 2015, TAP@STAF.

[18]  Pieter J. Mosterman,et al.  Requirements-Based Testing in Aircraft Control Design , 2005 .

[19]  Mihalis Yannakakis,et al.  Testing, optimization, and games , 2004, Proceedings of the 19th Annual IEEE Symposium on Logic in Computer Science, 2004..

[20]  Bernhard K. Aichernig,et al.  Killing strategies for model‐based mutation testing , 2015, Softw. Test. Verification Reliab..

[21]  Insup Lee,et al.  Specification-based testing with linear temporal logic , 2004, Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004..

[22]  Georgios E. Fainekos,et al.  Hybrid approximate gradient and stochastic descent for falsification of nonlinear systems , 2017, 2017 American Control Conference (ACC).

[23]  Sriram Sankaranarayanan,et al.  Falsification of temporal properties of hybrid systems using the cross-entropy method , 2012, HSCC '12.

[24]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[25]  Florian Lorber,et al.  Time for Mutants - Model-Based Mutation Testing with Timed Automata , 2013, TAP@STAF.

[26]  Kim G. Larsen,et al.  Cooperative Testing of Timed Systems , 2008, Electron. Notes Theor. Comput. Sci..

[27]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.