Statistical Model Checking Meets Property-Based Testing

In recent years, statistical model checking (SMC) has become increasingly popular, because it scales well to larger stochastic models and is relatively simple to implement. SMC solves the model checking problem by simulating the model for finitely many executions and uses hypothesis testing to infer if the samples provide statistical evidence for or against a property. Being based on simulation and statistics, SMC avoids the state-space explosion problem well-known from other model checking algorithms. In this paper we show how SMC can be easily integrated into a property-based testing framework, like FsCheck for C#. As a result we obtain a very flexible testing and simulation environment, where a programmer can define models and properties in a familiar programming language. The advantages: no external modelling language is needed and both stochastic models and implementations can be checked. In addition, we have access to the powerful test-data generators of a property-based testing tool. We demonstrate the feasibility of our approach by repeating three experiments from the SMC literature.

[1]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[2]  Marta Z. Kwiatkowska,et al.  Automated Verification of a Randomized Distributed Consensus Protocol Using Cadence SMV and PRISM , 2001, CAV.

[3]  Axel Legay,et al.  PLASMA-lab: A Flexible, Distributable Statistical Model Checking Library , 2013, QEST.

[4]  Bernhard K. Aichernig,et al.  Property-Based Testing with FsCheck by Deriving Properties from Business Rule Models , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[5]  David A. McAllester,et al.  Effective Bayesian Inference for Stochastic Programs , 1997, AAAI/IAAI.

[6]  Cyrille Jégourel,et al.  A Platform for High Performance Statistical Model Checking - PLASMA , 2012, TACAS.

[7]  P. McDermott-Wells,et al.  What is Bluetooth? , 2005, IEEE Potentials.

[8]  Kim G. Larsen,et al.  Statistical model checking for biological systems , 2015, International Journal on Software Tools for Technology Transfer.

[9]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[10]  J. Andel Sequential Analysis , 2022, The SAGE Encyclopedia of Research Design.

[11]  John Hughes,et al.  QuickCheck Testing for Fun and Profit , 2007, PADL.

[12]  Axel Legay,et al.  Statistical Model Checking with Change Detection , 2016, LNCS Trans. Found. Mastering Chang..

[13]  Konstantinos Sagonas,et al.  A PropEr integration of types and function specifications with property-based testing , 2011, Erlang Workshop.

[14]  W. Hoeffding Probability Inequalities for sums of Bounded Random Variables , 1963 .

[15]  Marta Kwiatkowska,et al.  Verifying Randomized Distributed Algorithms with PRISM , 2000 .

[16]  Maurice Herlihy,et al.  Fast Randomized Consensus Using Shared Memory , 1990, J. Algorithms.

[17]  Kim G. Larsen,et al.  UPPAAL-SMC: Statistical Model Checking for Priced Timed Automata , 2012, QAPL.

[18]  Marie Duflot,et al.  A formal analysis of bluetooth device discovery , 2006, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[19]  Robert M. Hierons,et al.  Generating Feasible Transition Paths for Testing from an Extended Finite State Machine (EFSM) , 2009, 2009 International Conference on Software Testing Verification and Validation.

[20]  Bernhard K. Aichernig,et al.  Towards integrating statistical model checking into property-based testing , 2016, 2016 ACM/IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE).

[21]  Thomas Hérault,et al.  Approximate Probabilistic Model Checking , 2004, VMCAI.

[22]  Mahesh Viswanathan,et al.  VESTA: A statistical model-checker and analyzer for probabilistic systems , 2005, Second International Conference on the Quantitative Evaluation of Systems (QEST'05).

[23]  Chung-Kil Hur,et al.  R2: An Efficient MCMC Sampler for Probabilistic Programs , 2014, AAAI.

[24]  Shigeru Kusakabe,et al.  Performance Evaluation of A Testing Framework Using QuickCheck and Hadoop , 2012, J. Inf. Process..

[25]  Matt P. Wand,et al.  Using Infer.NET for Statistical Analyses , 2011 .

[26]  Håkan L. S. Younes Ymer: A Statistical Model Checker , 2005, CAV.

[27]  Cyrille Jégourel Rare event simulation for statistical model checking , 2014 .

[28]  Axel Legay,et al.  Statistical Model Checking: An Overview , 2010, RV.

[29]  José Meseguer,et al.  PVeStA: A Parallel Statistical Model Checking and Quantitative Analysis Tool , 2011, CALCO.

[30]  Kim G. Larsen,et al.  Time for Statistical Model Checking of Real-Time Systems , 2011, CAV.

[31]  Thomas A. Henzinger,et al.  Probabilistic programming , 2014, FOSE.

[32]  Axel Legay,et al.  On Statistical Model Checking with PLASMA , 2014, 2014 Theoretical Aspects of Software Engineering Conference.

[33]  Axel Legay,et al.  Contracts and Behavioral Patterns for SoS: The EU IP DANSE approach , 2013, AiSoS.

[34]  Amir Pnueli,et al.  Verification of multiprocess probabilistic protocols , 1984, PODC '84.