Full version: An evaluation of estimation techniques for probabilistic reachability

We evaluate numerically-precise Monte Carlo (MC), Quasi-Monte Carlo (QMC) and Randomised Quasi-Monte Carlo (RQMC) methods for computing probabilistic reachability in hybrid systems with random parameters. Computing reachability probability amounts to computing (multidimensional) integrals. In particular, we pay attention to QMC methods due to their theoretical benefits in convergence speed with respect to the MC method. The Koksma-Hlawka inequality is a standard result that bounds the approximation of an integral by QMC techniques. However, it is not useful in practice because it depends on the variation of the integrand function, which is in general difficult to compute. The question arises whether it is possible to apply statistical or empirical methods for estimating the approximation error. In this paper we compare a number of interval estimation techniques based on the Central Limit Theorem (CLT), and we also introduce a new approach based on the CLT for computing confidence intervals for probability near the borders of the [0,1] interval. Based on our analysis, we provide justification for the use of the developed approach and suggest usage guidelines for probability estimation techniques.

[1]  Vivek Pradhan,et al.  Confidence Interval of the Difference of Two Independent Binomial Proportions Using Weighted Profile Likelihood , 2008, Commun. Stat. Simul. Comput..

[2]  Empirically estimating error of integration by quasi-Monte Carlo method , 2014 .

[3]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[4]  Bruno Tuffin.  Randomization of Quasi-Monte Carlo Methods for Error Estimation: Survey and Normal Approximation * , 2004, Monte Carlo Methods Appl..

[5]  Harald Niederreiter,et al.  Random number generation and Quasi-Monte Carlo methods , 1992, CBMS-NSF regional conference series in applied mathematics.

[6]  Thomas A. Henzinger,et al.  Hybrid Automata: An Algorithmic Approach to the Specification and Verification of Hybrid Systems , 1992, Hybrid Systems.

[7]  Kamaljit Kaur,et al.  Bayesian estimation for Gini index and a poverty measure in case of pareto distribution using Jeffreys' prior , 2015, Model. Assist. Stat. Appl..

[8]  Edmund M. Clarke,et al.  Delta-Decidability over the Reals , 2012, 2012 27th Annual IEEE Symposium on Logic in Computer Science.

[9]  E. B. Wilson.  Probable Inference, the Law of Succession, and Statistical Inference , 1927 .

[10]  Edmund M. Clarke,et al.  Bayesian statistical model checking with application to Stateflow/Simulink verification , 2013, Formal Methods Syst. Des..

[11]  A. Agresti,et al.  Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions , 1998 .

[12]  Edmund M. Clarke,et al.  dReal: An SMT Solver for Nonlinear Theories over the Reals , 2013, CADE.

[13]  F. J. Anscombe,et al.  THE TRANSFORMATION OF POISSON, BINOMIAL AND NEGATIVE-BINOMIAL DATA , 1948 .

[14]  Edmund M. Clarke,et al.  Satisfiability modulo ODEs , 2013, 2013 Formal Methods in Computer-Aided Design.

[15]  L. Brown,et al.  Interval Estimation for a Binomial Proportion , 2001 .

[16]  F. J. Anscombe,et al.  On estimating binomial response relations , 1956 .

[17]  C. Kemerer,et al.  Network Externalities in Microcomputer Software: An Econometric Analysis of the Spreadsheet Market , 2014 .

[18]  Paolo Zuliani,et al.  ProbReach: verified probabilistic delta-reachability for stochastic hybrid systems , 2014, HSCC.

[19]  Paolo Zuliani,et al.  Probabilistic Hybrid Systems Verification via SMT and Monte Carlo Techniques , 2016, Haifa Verification Conference.

[20]  E. S. Pearson,et al.  THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL , 1934 .

[21]  Henryk Wozniakowski,et al.  Information-based complexity , 1987, Nature.

[22]  M. Pagano,et al.  Evaluating Confidence Interval Methods for Binomial Proportions in Clustered Surveys , 2015 .

[23]  Anand Srivastav,et al.  Finding optimal volume subintervals with k points and calculating the star discrepancy are NP-hard problems , 2009, J. Complex..

[24]  Martin Fränzle,et al.  SAT Modulo ODE: A Direct SAT Approach to Hybrid Systems , 2008, ATVA.