The 2019 Comparison of Tools for the Analysis of Quantitative Formal Models - (QComp 2019 Competition Report)

Quantitative formal models capture probabilistic behaviour, real-time aspects, or general continuous dynamics. A number of tools support their automatic analysis with respect to dependability or performance properties. QComp 2019 is the first, friendly competition among such tools. It focuses on stochastic formalisms from Markov chains to probabilistic timed automata specified in the Jani model exchange format, and on probabilistic reachability, expected-reward, and steady-state properties. QComp draws its benchmarks from the new Quantitative Verification Benchmark Set. Participating tools, which include probabilistic model checkers and planners as well as simulation-based tools, are evaluated in terms of performance, versatility, and usability. In this paper, we report on the challenges in setting up a quantitative verification competition, present the results of QComp 2019, summarise the lessons learned, and provide an outlook on the features of the next edition of QComp.

[1]  Krishnendu Chatterjee,et al.  Verification of Markov Decision Processes Using Learning Algorithms , 2014, ATVA.

[2]  Holger Hermanns,et al.  Exploiting Robust Optimization for Interval Probabilistic Bisimulation , 2016, QEST.

[3]  Marta Z. Kwiatkowska,et al.  Game-based Abstraction for Markov Decision Processes , 2006, Third International Conference on the Quantitative Evaluation of Systems - (QEST'06).

[4]  Joost-Pieter Katoen,et al.  A compositional modelling and analysis framework for stochastic hybrid systems , 2012, Formal Methods in System Design.

[5]  Robert Givan,et al.  FF-Replan: A Baseline for Probabilistic Planning , 2007, ICAPS.

[6]  Pieter-Tjerk de Boer,et al.  Rare event simulation for dynamic fault trees , 2019, Reliab. Eng. Syst. Saf..

[7]  Blai Bonet,et al.  Faster Heuristic Search Algorithms for Planning with Uncertainty and Full Feedback , 2003, IJCAI.

[8]  Yi Li,et al.  iscasMc: A Web-Based Probabilistic Model Checker , 2014, FM.

[9]  Sean Sedwards,et al.  A Statistical Model Checker for Nondeterminism and Rare Events , 2018, TACAS.

[10]  Holger Hermanns,et al.  Explicit Model Checking of Very Large MDP Using Partitioning and Secondary Storage , 2015, ATVA.

[11]  Scott Sanner,et al.  A Survey of the Seventh International Planning Competition , 2012, AI Mag..

[12]  Benjamin Monmege,et al.  Interval iteration algorithm for MDPs and IMDPs , 2017, Theor. Comput. Sci..

[13]  Hector Geffner,et al.  Heuristic Search for Generalized Stochastic Shortest Path MDPs , 2011, ICAPS.

[14]  Holger Hermanns,et al.  Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes , 2017, QEST.

[15]  Thomas Hérault,et al.  Approximate Probabilistic Model Checking , 2004, VMCAI.

[16]  Malte Helmert,et al.  The Fast Downward Planning System , 2006, J. Artif. Intell. Res..

[17]  Holger Hermanns,et al.  Optimal Continuous Time Markov Decisions , 2015, ATVA.

[18]  Holger Hermanns,et al.  MODEST: A Compositional Modeling Formalism for Hard and Softly Timed Systems , 2006, IEEE Transactions on Software Engineering.

[19]  Dirk Beyer,et al.  Competition on Software Verification - (SV-COMP) , 2012, TACAS.

[20]  Håkan L. S. Younes,et al.  The First Probabilistic Track of the International Planning Competition , 2005, J. Artif. Intell. Res..

[21]  Dejan Nickovic,et al.  Monitoring Temporal Properties of Continuous Signals , 2004, FORMATS/FTRTFT.

[22]  Lijun Zhang,et al.  On Probabilistic Automata in Continuous Time , 2010, 2010 25th Annual IEEE Symposium on Logic in Computer Science.

[23]  Arnd Hartmanns,et al.  A Comparison of Time- and Reward-Bounded Probabilistic Model Checking Techniques , 2016, SETTA.

[24]  Sebastian Junges,et al.  PROPhESY: A PRObabilistic ParamEter SYnthesis Tool , 2015, CAV.

[25]  Holger Hermanns,et al.  Electronic Communications of the EASST Volume 70 ( 2014 ) Proceedings of the 14 th International Workshop on Automated Verification of Critical Systems ( AVoCS 2014 ) Reachability and Reward Checking for Stochastic Timed Automata , 2014 .

[26]  William H. Sanders,et al.  Möbius 2.3: An extensible tool for dependability, security, and performance evaluation of large and complex system models , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[27]  Jeremy Sproston.  Decidable Model Checking of Probabilistic Hybrid Automata , 2000, FTRTFT.

[28]  Antonio Puliafito,et al.  Principles of Performance and Reliability Modeling and Evaluation , 2016 .

[29]  Pedro R. D'Argenio,et al.  Better Automated Importance Splitting for Transient Rare Events , 2017, SETTA.

[30]  Holger Hermanns,et al.  The Modest Toolset: An Integrated Environment for Quantitative Modelling and Verification , 2014, TACAS.

[31]  Jan Kretínský,et al.  The Hanoi Omega-Automata Format , 2015, CAV.

[32]  Thomas A. Henzinger,et al.  Probabilistic programming , 2014, FOSE.

[33]  Marta Z. Kwiatkowska,et al.  Performance analysis of probabilistic timed automata using digital clocks , 2003, Formal Methods Syst. Des..

[34]  Taolue Chen,et al.  Automatic verification of competitive stochastic systems , 2012, Formal Methods in System Design.

[35]  Lijun Zhang,et al.  Model Checking Probabilistic Epistemic Logic for Probabilistic Multiagent Systems , 2018, IJCAI.

[36]  Mariëlle Stoelinga,et al.  DFTCalc: a tool for efficient fault tree analysis (extended version) , 2013 .

[37]  Kousha Etessami,et al.  Multi-Objective Model Checking of Markov Decision Processes , 2007, Log. Methods Comput. Sci..

[38]  Christel Baier,et al.  Approximate Symbolic Model Checking of Continuous-Time Markov Chains , 1999, CONCUR.

[39]  Sven Schewe,et al.  Accelerated Model Checking of Parametric Markov Chains , 2018, ATVA.

[40]  R. Khan,et al.  Sequential Tests of Statistical Hypotheses. , 1972 .

[41]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[42]  Nils Jansen,et al.  Fast Debugging of PRISM Models , 2014, ATVA.

[43]  Blai Bonet,et al.  Labeled RTDP: Improving the Convergence of Real-Time Dynamic Programming , 2003, ICAPS.

[44]  Bernhard Steffen,et al.  RERS 2018: CTL, LTL, and Reachability , 2018, ISoLA.

[45]  Lijun Zhang,et al.  An Efficient Synthesis Algorithm for Parametric Markov Chains Against Linear Time Properties , 2016, SETTA.

[46]  Lijun Zhang,et al.  Measurability and safety verification for stochastic hybrid systems , 2011, HSCC '11.

[47]  Gerardo Rubino,et al.  Rare Event Simulation using Monte Carlo Methods , 2009 .

[48]  Christel Baier,et al.  Ensuring the Reliability of Your Model Checker: Interval Iteration for Markov Decision Processes , 2017, CAV.

[49]  Sebastian Junges,et al.  Fast Dynamic Fault Tree Analysis by Model Checking Techniques , 2018, IEEE Transactions on Industrial Informatics.

[50]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[51]  M. Okamoto.  Some inequalities relating to the partial sum of binomial probabilities , 1959 .

[52]  Marta Z. Kwiatkowska,et al.  PRISM-games: verification and strategy synthesis for stochastic multi-player games with multiple objectives , 2017, International Journal on Software Tools for Technology Transfer.

[53]  Christel Baier,et al.  Principles of model checking , 2008 .

[54]  Holger Hermanns,et al.  Flexible support for time and costs in scenario-aware dataflow , 2016, 2016 International Conference on Embedded Software (EMSOFT).

[55]  Sean Sedwards,et al.  Lightweight Statistical Model Checking in Nondeterministic Continuous Time , 2018, ISoLA.

[56]  Jane Hillston,et al.  A compositional approach to performance modelling , 1996 .

[57]  Arnd Hartmanns,et al.  The Quantitative Verification Benchmark Set , 2019, TACAS.

[58]  Sebastian Junges,et al.  Markov automata with multiple objectives , 2017, Formal Methods in System Design.

[59]  Olivier Roussel,et al.  The International SAT Solver Competitions , 2012, AI Mag..

[60]  Joost-Pieter Katoen,et al.  Sound Value Iteration , 2018, CAV.

[61]  Yuan Feng,et al.  Model Checking Omega-regular Properties for Quantum Markov Chains , 2017, CONCUR.

[62]  Sebastian Junges,et al.  A Storm is Coming: A Modern Probabilistic Model Checker , 2017, CAV.

[63]  Marco Beccuti,et al.  30 Years of GreatSPN , 2016 .

[64]  Patrik Haslum,et al.  New Admissible Heuristics for Domain-Independent Planning , 2005, AAAI.

[65]  Sebastian Junges,et al.  JANI: Quantitative Model and Tool Interaction , 2017, TACAS.

[66]  Pieter-Tjerk de Boer,et al.  Path-ZVA: general, efficient and automated importance sampling for highly reliable Markovian systems , 2018, 1806.11255.

[67]  Lijun Zhang,et al.  A Comparative Study of BDD Packages for Probabilistic Symbolic Model Checking , 2015, SETTA.

[68]  Håkan L. S. Younes,et al.  Probabilistic Verification of Discrete Event Systems Using Acceptance Sampling , 2002, CAV.

[69]  David Coppit,et al.  The Galileo fault tree analysis tool , 1999, Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No.99CB36352).

[70]  Lijun Zhang,et al.  A Simple Algorithm for Solving Qualitative Probabilistic Parity Games , 2016, CAV.

[71]  Marta Z. Kwiatkowska,et al.  Automatic verification of real-time systems with discrete probability distributions , 1999, Theor. Comput. Sci..

[72]  Sebastian Junges,et al.  Multi-cost Bounded Reachability in MDP , 2018, TACAS.

[73]  Christel Baier,et al.  Computing Quantiles in Markov Reward Models , 2013, FoSSaCS.

[74]  Sebastian Junges,et al.  Parameter Synthesis for Markov Models: Faster Than Ever , 2016, ATVA.

[75]  Jörg Hoffmann,et al.  Compiling Probabilistic Model Checking into Probabilistic Planning , 2018, ICAPS.

[76]  Olivier Buffet,et al.  Goal Probability Analysis in Probabilistic Planning: Exploring and Enhancing the State of the Art , 2016, J. Artif. Intell. Res..