MOMDPs: A Solution for Modelling Adaptive Management Problems

In conservation biology and natural resource management, adaptive management is an iterative process of improving management by reducing uncertainty via monitoring. Adaptive management is the principal tool for conserving endangered species under global change, yet adaptive management problems suffer from a poor suite of solution methods. The common approach used to solve an adaptive management problem is to assume the system state is known and the system dynamics can be one of a set of pre-defined models. The solution method used is unsatisfactory, employing value iteration on a discretized belief MDP which restricts the study to very small problems. We show how to overcome this limitation by modelling an adaptive management problem as a restricted Mixed Observability MDP called hidden model MDP (hmMDP). We demonstrate how to simplify the value function, the backup operator and the belief update computation. We show that, although a simplified case of POMDPs, hm-MDPs are PSPACE-complete in the finite-horizon case. We illustrate the use of this model to manage a population of the threatened Gouldian finch, a bird species endemic to Northern Australia. Our simple modelling approach is an important step towards efficient algorithms for solving adaptive management problems.

[1]  E. J. Sondik,et al.  The Optimal Control of Partially Observable Markov Decision Processes. , 1971 .

[2]  Carl J. Walters,et al.  ECOLOGICAL OPTIMIZATION AND ADAPTIVE MANAGEMENT , 1978 .

[3]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[4]  William S. Lovejoy,et al.  Computationally Feasible Bounds for Partially Observed Markov Decision Processes , 1991, Oper. Res..

[5]  Craig Boutilier,et al.  Computing Optimal Policies for Partially Observable Decision Processes Using Compact Representations , 1996, AAAI/IAAI, Vol. 2.

[6]  Ronen I. Brafman,et al.  A Heuristic Variable Grid Solution Method for POMDPs , 1997, AAAI/IAAI.

[7]  A. Cassandra,et al.  Exact and approximate algorithms for partially observable markov decision processes , 1998 .

[8]  Eric A. Hansen,et al.  An Improved Grid-Based Approximation Algorithm for POMDPs , 2001, IJCAI.

[9]  Blai Bonet,et al.  An epsilon-Optimal Grid-Based Algorithm for Partially Observable Markov Decision Processes , 2002, ICML.

[10]  Blai Bonet.  An 2-Optimal Grid-Based Algorithm for Partially Observable Markov Decision Processes , 2002 .

[11]  Joelle Pineau,et al.  Point-based value iteration: An anytime algorithm for POMDPs , 2003, IJCAI.

[12]  Vincent Conitzer,et al.  Definition and Complexity of Some Basic Metareasoning Problems , 2003, IJCAI.

[13]  Anne Condon,et al.  On the undecidability of probabilistic planning and related stochastic optimization problems , 2003, Artif. Intell..

[14]  Pat Langley,et al.  Editorial: On Machine Learning , 1986, Machine Learning.

[15]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[16]  Nikos A. Vlassis,et al.  Perseus: Randomized Point-based Value Iteration for POMDPs , 2005, J. Artif. Intell. Res..

[17]  P. Poupart.  Exploiting structure to efficiently solve large scale partially observable Markov decision processes , 2005 .

[18]  Jesse Hoey,et al.  An analytic solution to discrete Bayesian reinforcement learning , 2006, ICML.

[19]  I. Chades,et al.  When to stop managing or surveying cryptic threatened species , 2008, Proceedings of the National Academy of Sciences.

[20]  Byron K. Williams,et al.  Markov decision processes in natural resources management: Observability and uncertainty , 2009 .

[21]  Olivier Buffet,et al.  A Closer Look at MOMDPs , 2010, 2010 22nd IEEE International Conference on Tools with Artificial Intelligence.

[22]  David Hsu,et al.  Planning under Uncertainty for Robotic Tasks with Mixed Observability , 2010, Int. J. Robotics Res..

[23]  Eve McDonald-Madden,et al.  Uncertainty and adaptive management for biodiversity conservation , 2011 .

[24]  I. Chades,et al.  General rules for managing and surveying networks of pests, diseases, and endangered species , 2011, Proceedings of the National Academy of Sciences.