Counterexample-Guided Abstraction Refinement for POMDPs

Partially Observable Markov Decision Process (POMDP) is widely used to model probabilistic behavior for complex systems. Compared with MDPs, POMDP models a system more accurate but solving a POMDP generally takes exponential time in the size of its state space. This makes the formal verification and synthesis problems much more challenging for POMDPs, especially when multiple system components are involved. As a promising technique to reduce the verification complexity, the abstraction method tries to find an abstract system with a smaller state space but preserves enough properties for the verification purpose. While abstraction based verification has been explored extensively for MDPs, in this paper, we present the first result of POMDP abstraction and its refinement techniques. The main idea follows the counterexample-guided abstraction refinement (CEGAR) framework. Starting with a coarse guess for the POMDP abstraction, we iteratively use counterexamples from formal verification to refine the abstraction until the abstract system can be used to infer the verification result for the original POMDP. Our main contributions have two folds: 1) we propose a novel abstract system model for POMDP and a new simulation relation to capture the partial observability then prove the preservation on a fragment of Probabilistic Computation Tree Logic (PCTL); 2) to find a proper abstract system that can prove or disprove the satisfaction relation on the concrete POMDP, we develop a novel refinement algorithm. Our work leads to a sound and complete CEGAR framework for POMDP.

[1]  Edmund M. Clarke,et al.  Assume-Guarantee Abstraction Refinement for Probabilistic Systems , 2012, CAV.

[2]  Lu Feng,et al.  Automated Learning of Probabilistic Assumptions for Compositional Reasoning , 2011, FASE.

[3]  Judy Goldsmith,et al.  Complexity issues in Markov decision processes , 1998, Proceedings. Thirteenth Annual IEEE Conference on Computational Complexity (Formerly: Structure in Complexity Theory Conference) (Cat. No.98CB36247).

[4]  Joel Veness,et al.  Monte-Carlo Planning in Large POMDPs , 2010, NIPS.

[5]  Karl Johan Åström,et al.  Optimal control of Markov processes with incomplete state information , 1965 .

[6]  Krishnendu Chatterjee,et al.  A survey of partial-observation stochastic parity games , 2012, Formal Methods in System Design.

[7]  Joost-Pieter Katoen,et al.  Counterexamples in Probabilistic Model Checking , 2007, TACAS.

[8]  Anne Condon,et al.  On the Undecidability of Probabilistic Planning and Infinite-Horizon Partially Observable Markov Decision Problems , 1999, AAAI/IAAI.

[9]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching Time Temporal Logic , 2008, 25 Years of Model Checking.

[10]  Lu Feng,et al.  Compositional Verification of Probabilistic Systems Using Learning , 2010, 2010 Seventh International Conference on the Quantitative Evaluation of Systems.

[11]  Z. Caner Taşkın.  Tutorial Guide to Mixed-Integer Programming Models and Solution Techniques , 2008 .

[12]  Marta Z. Kwiatkowska,et al.  Stochastic Model Checking , 2007, SFM.

[13]  Hai Lin,et al.  Assume-guarantee reasoning framework for MDP-POMDP , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[14]  Rangoli Sharan,et al.  Formal Methods for Control Synthesis in Partially Observed Environments: Application to Autonomous Robotic Manipulation , 2014 .

[15]  Christel Baier,et al.  Deciding Bisimilarity and Similarity for Probabilistic Processes , 2000, J. Comput. Syst. Sci..

[16]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[17]  Lijun Zhang,et al.  Learning Weighted Assumptions for Compositional Verification of Markov Decision Processes , 2016, ACM Trans. Softw. Eng. Methodol..

[18]  Lijun Zhang,et al.  Probabilistic CEGAR , 2008, CAV.

[19]  Hai Lin,et al.  Learning based supervisor synthesis of POMDP for PCTL specifications , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[20]  Weihong Zhang,et al.  Speeding Up the Convergence of Value Iteration in Partially Observable Markov Decision Processes , 2011, J. Artif. Intell. Res..

[21]  Joelle Pineau,et al.  Anytime Point-Based Approximations for Large POMDPs , 2006, J. Artif. Intell. Res..

[22]  Thomas A. Henzinger,et al.  Abstractions from proofs , 2004, POPL.

[23]  Joost-Pieter Katoen,et al.  Counterexample Generation in Probabilistic Model Checking , 2009, IEEE Transactions on Software Engineering.

[24]  Hongyang Qu,et al.  Compositional probabilistic verification through multi-objective model checking , 2013, Inf. Comput..

[25]  Christel Baier,et al.  Principles of model checking , 2008 .

[26]  Lu Feng,et al.  Learning-Based Compositional Verification for Synchronous Probabilistic Systems , 2011, ATVA.

[27]  C. M. Parkes,et al.  Seventh International Conference , 2005 .

[28]  Anders P. Ravn,et al.  Proceedings of the 7th International Symposium on Automated Technology for Verification and Analysis , 2009 .

[29]  Nils Jansen,et al.  The COMICS Tool - Computing Minimal Counterexamples for DTMCs , 2012, ATVA.

[30]  Hongyang Qu,et al.  Assume-Guarantee Verification for Probabilistic Systems , 2010, TACAS.

[31]  Csaba Szepesvári,et al.  Bandit Based Monte-Carlo Planning , 2006, ECML.

[32]  Nancy A. Lynch,et al.  Probabilistic Simulations for Probabilistic Processes , 1994, Nord. J. Comput..

[33]  Jan J. M. M. Rutten,et al.  Mathematical techniques for analyzing concurrent and probabilistic systems , 2004, CRM monograph series.

[34]  Krishnendu Chatterjee,et al.  What is decidable about partially observable Markov decision processes with ω-regular objectives , 2013, J. Comput. Syst. Sci..

[35]  Krishnendu Chatterjee,et al.  Randomness for Free , 2010, MFCS.