Three New Algorithms to Solve N-POMDPs

In many fields in computational sustainability, applications of POMDPs are inhibited by the complexity of the optimal solution. One way of delivering simple solutions is to represent the policy with a small number of α-vectors. We would like to find the best possible policy that can be expressed using a fixed number N of α-vectors. We call this the N-POMDP problem. The existing solver α-min approximately solves finite-horizon POMDPs with a controllable number of α-vectors. However α-min is a greedy algorithm without performance guarantees, and it is rather slow. This paper proposes three new algorithms, based on a general approach that we call α-min-2. These three algorithms are able to approximately solve N-POMDPs. α-min-2-fast (heuristic) and α-min-2-p (with performance guarantees) are designed to complement an existing POMDP solver, while α-min-2-solve (heuristic) is a solver itself. Complexity results are provided for each of the algorithms, and they are tested on well-known benchmarks. These new algorithms will help users to interpret solutions to POMDP problems in computational sustainability.

[1]  Karl Johan Åström,et al.  Optimal control of Markov processes with incomplete state information , 1965 .

[2]  Edward J. Sondik,et al.  The Optimal Control of Partially Observable Markov Processes over a Finite Horizon , 1973, Oper. Res..

[3]  Teofilo F. GONZALEZ,et al.  Clustering to Minimize the Maximum Intercluster Distance , 1985, Theor. Comput. Sci..

[4]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[5]  Hsien-Te Cheng,et al.  Algorithms for partially observable markov decision processes , 1989 .

[6]  M. Littman,et al.  Efficient dynamic-programming updates in partially observable Markov decision processes , 1995 .

[7]  Leslie Pack Kaelbling,et al.  Planning and Acting in Partially Observable Stochastic Domains , 1998, Artif. Intell..

[8]  Anne Condon,et al.  On the undecidability of probabilistic planning and related stochastic optimization problems , 2003, Artif. Intell..

[9]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[10]  Nikos A. Vlassis,et al.  Perseus: Randomized Point-based Value Iteration for POMDPs , 2005, J. Artif. Intell. Res..

[11]  Joelle Pineau,et al.  Anytime Point-Based Approximations for Large POMDPs , 2006, J. Artif. Intell. Res..

[12]  David Hsu,et al.  SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces , 2008, Robotics: Science and Systems.

[13]  I. Chades,et al.  When to stop managing or surveying cryptic threatened species , 2008, Proceedings of the National Academy of Sciences.

[14]  Olivier Buffet,et al.  Markov Decision Processes in Artificial Intelligence , 2010 .

[15]  I. Chades,et al.  Allocating conservation resources between areas where persistence of a species is uncertain. , 2011, Ecological applications : a publication of the Ecological Society of America.

[16]  I. Chades,et al.  Optimally managing under imperfect detection: A method for plant invasions , 2011 .

[17]  Kee-Eung Kim,et al.  Closing the Gap: Improved Bounds on Optimal POMDP Solutions , 2011, ICAPS.

[18]  Guy Shani,et al.  Noname manuscript No. (will be inserted by the editor) A Survey of Point-Based POMDP Solvers , 2022 .

[19]  I. Chades,et al.  Which States Matter? An Application of an Intelligent Discretization Method to Solve a Continuous POMDP in Conservation Biology , 2012, PloS one.

[20]  Michael R. Springborn,et al.  A density projection approach for non-trivial information dynamics: Adaptive management of stochastic natural resources , 2013 .

[21]  Olivier Buffet,et al.  Adaptive Management of Migratory Birds Under Sea Level Rise , 2013, IJCAI.

[22]  Krishna Pacifici,et al.  Addressing structural and observational uncertainty in resource management. , 2014, Journal of environmental management.

[23]  Hugh P. Possingham,et al.  Frontiers inEcology and the Environment Why do we map threats ? Linking threat mapping with actions to make better conservation decisions , 2015 .

[24]  I. Chades,et al.  Adapting environmental management to uncertain but inevitable change , 2015, Proceedings of the Royal Society B: Biological Sciences.

[25]  Thomas G. Dietterich,et al.  α-min: A Compact Approximate Solver For Finite-Horizon POMDPs , 2015, IJCAI.