Monte-Carlo Policy Synthesis in POMDPs with Quantitative and Qualitative Objectives

Autonomous robots operating in uncertain environments often face the problem of planning under a mix of formal, qualitative requirements, for example the assertion that the robot reaches a goal location safely, and optimality criteria, for example that the path to the goal is as short or energy-efficient as possible. Such problems can be modeled as Partially Observable Markov Decision Processes (POMDPs) with quantitative and qualitative objectives. In this paper, we present a new policy synthesis algorithm, called Policy Synthesis with Statistical Model Checking (PO-SMC), for such POMDPs. While previous policy synthesis approaches for this setting use symbolic tools (for example satisfiability solvers) to meet the qualitative requirements, our approach is based on Monte Carlo sampling and uses statistical model checking to ensure that the qualitative requirements are satisfied with high confidence. An appeal of statistical model checking is that it can handle rich temporal requirements such as safe-reachability, while being far more scalable than symbolic methods. The safe-reachability property combines the safety and reachability requirements as a single qualitative requirement. While our use of sampling introduces approximations that symbolic approaches do not require, we present theoretical results that show that the error due to approximation is bounded. Our experimental results demonstrate that PO-SMC consistently performs orders of magnitude faster than existing symbolic methods for policy synthesis under qualitative and quantitative requirements.