Long-Run Rewards for Markov Automata

Markov automata are a powerful formalism for modelling systems which exhibit nondeterminism, probabilistic choices and continuous stochastic timing. We consider the computation of long-run average rewards, the most classical problem in continuous-time Markov model analysis. We propose an algorithm based on value iteration. It improves the state of the art by orders of magnitude. The contribution is rooted in a fresh look on Markov automata, namely by treating them as an efficient encoding of CTMDPs with ---i?źin the worst casei?ź--- exponentially more transitions.

[1]  I. G. MacKenzie,et al.  Stochastic Processes with Applications , 1992 .

[2]  GhemawatSanjay,et al.  The Google file system , 2003 .

[3]  Roland Wunderling,et al.  Paralleler und objektorientierter Simplex-Algorithmus , 1996 .

[4]  Krishnendu Chatterjee,et al.  Faster and dynamic algorithms for maximal end-component decomposition and related graph problems in probabilistic verification , 2011, SODA '11.

[5]  Joost-Pieter Katoen,et al.  Modelling, Reduction and Analysis of Markov Automata (extended version) , 2013, QEST.

[6]  Dennis Guck,et al.  Quantitative Analysis of Markov Automata , 2012 .

[7]  Mariëlle Stoelinga,et al.  Modelling and Analysis of Markov Reward Automata , 2014, ATVA.

[8]  Joost-Pieter Katoen,et al.  On the use of model checking techniques for dependability evaluation , 2000, Proceedings 19th IEEE Symposium on Reliable Distributed Systems SRDS-2000.

[9]  Joost-Pieter Katoen,et al.  Analysis of Timed and Long-Run Objectives for Markov Automata , 2014, Log. Methods Comput. Sci..

[10]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[11]  William J. Stewart,et al.  Introduction to the numerical solution of Markov Chains , 1994 .

[12]  Mark Timmer.  SCOOP: A Tool for SymboliC Optimisations of Probabilistic Processes , 2011, 2011 Eighth International Conference on Quantitative Evaluation of SysTems.

[13]  Lijun Zhang,et al.  On Probabilistic Automata in Continuous Time , 2010, 2010 25th Annual IEEE Symposium on Logic in Computer Science.

[14]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[15]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[16]  Holger Hermanns,et al.  Model Checking Algorithms for Markov Automata , 2012, Electron. Commun. Eur. Assoc. Softw. Sci. Technol..