Falsification of temporal properties of hybrid systems using the cross-entropy method

Randomized testing is a popular approach for checking properties of large embedded system designs. It is well known that a uniform random choice of test inputs is often sub-optimal. Ideally, the choice of inputs has to be guided by choosing the right input distributions in order to expose corner-case violations. However, this is also known to be a hard problem, in practice. In this paper, we present an application of the cross-entropy method for adaptively choosing input distributions for falsifying temporal logic properties of hybrid systems. We present various choices for representing input distribution families for the cross-entropy method, ranging from a complete partitioning of the input space into cells to a factored distribution of the input using graphical models. Finally, we experimentally compare the falsification approach using the cross-entropy method to other stochastic and heuristic optimization techniques implemented inside the tool S-Taliro over a set of benchmark systems. The performance of the cross entropy method is quite promising. We find that sampling inputs using the cross-entropy method guided by trace robustness can discover violations faster, and more consistently than the other competing methods considered.

[1]  Antoine Girard,et al.  Temporal Logic Verific ation Using Simulation , 2006 .

[2]  James A. Bucklew,et al.  Introduction to Rare Event Simulation , 2010 .

[3]  J. Hammersley SIMULATION AND THE MONTE CARLO METHOD , 1982 .

[4]  Ron Koymans,et al.  Specifying real-time properties with metric temporal logic , 1990, Real-Time Systems.

[5]  Antoine Girard,et al.  Verification Using Simulation , 2006, HSCC.

[6]  Thomas A. Henzinger,et al.  The Algorithmic Analysis of Hybrid Systems , 1995, Theor. Comput. Sci..

[7]  Dirk P. Kroese,et al.  The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning , 2004 .

[8]  Lih-Yuan Deng,et al.  The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation, and Machine Learning , 2006, Technometrics.

[9]  Lydia E. Kavraki,et al.  Hybrid Systems: From Verification to Falsification , 2007, CAV.

[10]  Edmund M. Clarke,et al.  Statistical Model Checking for Cyber-Physical Systems , 2011, ATVA.

[11]  Qianchuan Zhao,et al.  Generating test inputs for embedded control systems , 2003 .

[12]  Radu Grosu,et al.  Monte Carlo Model Checking , 2005, TACAS.

[13]  Joshua A. Levine,et al.  Sampling-based planning, control and verification of hybrid systems , 2000 .

[14]  Sriram Sankaranarayanan,et al.  Monte-carlo techniques for falsification of temporal properties of non-linear hybrid systems , 2010, HSCC '10.

[15]  Tarik Nahhal,et al.  Test Coverage for Continuous and Hybrid Systems , 2007, CAV.

[16]  Insup Lee,et al.  Robust Test Generation and Coverage for Hybrid Systems , 2007, HSCC.

[17]  Antoine Girard,et al.  Temporal Logic Verification Using Simulation , 2006, FORMATS.

[18]  Håkan L. S. Younes,et al.  Statistical probabilistic model checking with a focus on time-bounded properties , 2006, Inf. Comput..

[19]  Vijay Kumar,et al.  Adaptive RRTs for Validating Hybrid Robotic Control Systems , 2004, WAFR.

[20]  M. Fisher,et al.  A semiclosed-loop algorithm for the control of blood glucose levels in diabetics , 1991, IEEE Transactions on Biomedical Engineering.

[21]  Magnus Egerstedt,et al.  Control Theoretic Splines: Optimal Control, Statistics, and Path Planning , 2009 .

[22]  Olaf Stursberg,et al.  On Systematic Simulation of Open Continuous Systems , 2003, HSCC.

[23]  Eitan Farchi,et al.  Cross-Entropy-Based Replay of Concurrent Programs , 2009, FASE.

[24]  François Fages,et al.  On a Continuous Degree of Satisfaction of Temporal Logic Formulae with Applications to Systems Biology , 2008, CMSB.

[25]  Eitan Farchi,et al.  Cross-Entropy Based Testing , 2007, Formal Methods in Computer Aided Design (FMCAD'07).

[26]  Emilio Frazzoli,et al.  Incremental Search Methods for Reachability Analysis of Continuous and Hybrid Systems , 2004, HSCC.

[27]  John Lygeros,et al.  On reachability and minimum cost optimal control , 2004, Autom..

[28]  Dirk P. Kroese,et al.  Simulation and the Monte Carlo Method (Wiley Series in Probability and Statistics) , 1981 .

[29]  Dirk P. Kroese,et al.  Cross‐Entropy Method , 2011 .

[30]  Oded Maler,et al.  Sensitive state-space exploration , 2008, 2008 47th IEEE Conference on Decision and Control.

[31]  Sriram Sankaranarayanan,et al.  S-TaLiRo: A Tool for Temporal Logic Falsification for Hybrid Systems , 2011, TACAS.

[32]  Oded Maler,et al.  Verification of Analog and Mixed-Signal Circuits Using Hybrid System Techniques , 2004, FMCAD.

[33]  Lydia E. Kavraki,et al.  Falsification of LTL Safety Properties in Hybrid Systems , 2009, TACAS.

[34]  Oded Maler,et al.  Systematic Simulation Using Sensitivity Analysis , 2007, HSCC.

[35]  Sriram Sankaranarayanan,et al.  State space exploration using feedback constraint generation and Monte-Carlo sampling , 2007, ESEC-FSE '07.

[36]  B. I. Silva,et al.  Modeling and Verifying Hybrid Dynamic Systems Using CheckMate , 2001 .

[37]  Pravin Varaiya,et al.  Modeling and verification of hybrid systems , 1995, Proceedings of 1995 American Control Conference - ACC'95.

[38]  M. Branicky,et al.  Sampling-Based Planning and Control , 2003 .

[39]  Axel Legay,et al.  Statistical Model Checking of Mixed-Analog Circuits with an Application to a Third Order Delta-Sigma Modulator , 2009, Haifa Verification Conference.

[40]  Dirk P. Kroese,et al.  The Cross Entropy Method: A Unified Approach To Combinatorial Optimization, Monte-carlo Simulation (Information Science and Statistics) , 2004 .

[41]  Edmund M. Clarke,et al.  Verification of Supervisory Control Software Using State Proximity and Merging , 2008, HSCC.

[42]  Yashwanth Singh Rahul Annapureddy,et al.  Ant colonies for Temporal Logic falsification of hybrid systems , 2010, IECON 2010 - 36th Annual Conference on IEEE Industrial Electronics Society.