MDPtoolbox: a multi-platform toolbox to solve stochastic dynamic programming problems

Stochastic dynamic programming (SDP) or Markov decision processes (MDP) are increasingly being used in ecology to find the best decisions over time and under uncertainty so that the chance of achieving an objective is maximised. To date, few programs are available to solve SDP/MDP. We present MDPtoolbox, a multi-platform set of functions to solve Markov decision problems (MATLAB, GNU Octave, Scilab and R). MDPtoolbox provides state-of-the-art and ready to use algorithms to solve a wide range of MDPs. MDPtoolbox is easy to use, freely available and has been continuously improved since 2004. We illustrate how to use MDPtoolbox on a dynamic reserve design problem.

[1]  Byron K. Williams,et al.  Markov decision processes in natural resources management: Observability and uncertainty , 2009 .

[2]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[3]  Lucile Marescot,et al.  Complex decisions made simple: a primer on stochastic dynamic programming , 2013 .

[4]  Nathalie Peyrard,et al.  A framework and a mean-field algorithm for the local control of spatial processes , 2012, Int. J. Approx. Reason..

[5]  C. Clark,et al.  Dynamic Modeling in Behavioral Ecology , 2019 .

[6]  François Charpillet,et al.  Dynamic optimization over infinite-time horizon: web-building strategy in an orb-weaving spider as a case study. , 2006, Journal of theoretical biology.

[7]  Hugh P. Possingham,et al.  Prioritizing global conservation efforts , 2006, Nature.

[8]  S. Polasky,et al.  Dynamic reserve site selection , 2004 .

[9]  Olivier Buffet,et al.  MOMDPs: A Solution for Modelling Adaptive Management Problems , 2012, AAAI.

[10]  Olivier Buffet,et al.  Markov decision processes in artificial intelligence : MDPs, beyond MDPs and applications , 2010 .

[11]  I. Chades,et al.  Simple rules to contain an invasive species with a complex life cycle and high dispersal capacity , 2012 .

[12]  Cindy E. Hauser,et al.  Experimental or precautionary? Adaptive management over a range of time horizons , 2007 .

[13]  Hugh P. Possingham,et al.  A decision framework for management of conflicting production and biodiversity goals for a commercially valuable invasive species , 2014 .

[14]  I. Chades,et al.  Beyond stochastic dynamic programming: a heuristic sampling method for optimizing conservation decisions in very large state spaces , 2011 .

[15]  Hugh P. Possingham,et al.  The business of biodiversity: Applying decision theory principles to nature conservation , 2001 .

[16]  Régis Sabbadin,et al.  Dynamic reserve site selection under contagion risk of deforestation , 2007 .

[17]  Carl J. Walters,et al.  ECOLOGICAL OPTIMIZATION AND ADAPTIVE MANAGEMENT , 1978 .

[18]  Nenad Mladenovic,et al.  A variable neighborhood search based algorithm for finite-horizon Markov Decision Processes , 2010, Appl. Math. Comput..

[19]  Olivier Buffet,et al.  Adaptive Management of Migratory Birds Under Sea Level Rise , 2013, IJCAI.

[20]  A. Houston,et al.  Optimal daily routines of singing and foraging in a bird singing to attract a mate , 2004, Behavioral Ecology and Sociobiology.

[21]  Ann Gordon-Ross,et al.  An MDP-Based Dynamic Optimization Methodology for Wireless Sensor Networks , 2012, IEEE Transactions on Parallel and Distributed Systems.

[22]  I. Chades,et al.  Setting Realistic Recovery Targets for Two Interacting Endangered Species, Sea Otter and Northern Abalone , 2012, Conservation biology : the journal of the Society for Conservation Biology.

[23]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[24]  I. Chades,et al.  General rules for managing and surveying networks of pests, diseases, and endangered species , 2011, Proceedings of the National Academy of Sciences.