Complex decisions made simple: a primer on stochastic dynamic programming

Summary Under increasing environmental and financial constraints, ecologists are faced with making decisions about dynamic and uncertain biological systems. To do so, stochastic dynamic programming (SDP) is the most relevant tool for determining an optimal sequence of decisions over time. Despite an increasing number of applications in ecology, SDP still suffers from a lack of widespread understanding. The required mathematical and programming knowledge as well as the absence of introductory material provide plausible explanations for this. Here, we fill this gap by explaining the main concepts of SDP and providing useful guidelines to implement this technique, including R code. We illustrate each step of SDP required to derive an optimal strategy using a wildlife management problem of the French wolf population. Stochastic dynamic programming is a powerful technique to make decisions in presence of uncertainty about biological stochastic systems changing through time. We hope this review will provide an entry point into the technical literature about SDP and will improve its application in ecology.

[1]  William J. Reed,et al.  Optimal escapement levels in stochastic and deterministic harvesting models , 1979 .

[2]  Jay Sullivan,et al.  Optimal forest harvest decisions: A stochastic dynamic programming approach , 1993 .

[3]  Peter W J Baxter,et al.  Optimal eradication: when to stop looking for an invasive plant. , 2006, Ecology letters.

[4]  Cindy E. Hauser,et al.  Experimental or precautionary? Adaptive management over a range of time horizons , 2007 .

[5]  C. Clark,et al.  Dynamic State Variable Models in Ecology: Methods and Applications , 2001 .

[6]  D. Ludwig,et al.  Life-History Strategies for Energy Gain and Predator Avoidance Under Time Constraints , 1990, The American Naturalist.

[7]  Hugh P. Possingham,et al.  OPTIMAL FIRE MANAGEMENT FOR MAINTAINING COMMUNITY DIVERSITY , 1999 .

[8]  Michael C. Runge,et al.  THE IMPORTANCE OF FUNCTIONAL FORM IN OPTIMAL CONTROL SOLUTIONS OF PROBLEMS IN POPULATION DYNAMICS , 2002 .

[9]  C. Clark,et al.  Dynamic Modeling in Behavioral Ecology , 2019 .

[10]  I. Chades,et al.  Beyond stochastic dynamic programming: a heuristic sampling method for optimizing conservation decisions in very large state spaces , 2011 .

[11]  H. Simon.  Rational Decision Making in Business Organizations , 1978 .

[12]  Martin J. Folk,et al.  A matter of tradeoffs: Reintroduction as a multiple objective decision , 2013 .

[13]  Hugh P. Possingham,et al.  Prioritizing global conservation efforts , 2006, Nature.

[14]  C. T. Moore,et al.  Uncertainty and the management of mallard harvests , 1997 .

[15]  M. Conroy,et al.  Analysis and Management of Animal Populations , 2002 .

[16]  Stephens,et al.  Consequences of the Allee effect for behaviour, ecology and conservation. , 1999, Trends in ecology & evolution.

[17]  I. Chades,et al.  When to stop managing or surveying cryptic threatened species , 2008, Proceedings of the National Academy of Sciences.

[18]  Michael R. Springborn,et al.  A density projection approach for non-trivial information dynamics: Adaptive management of stochastic natural resources , 2013 .

[19]  Julien Martin,et al.  Structured decision making as a proactive approach to dealing with sea level rise in Florida , 2011 .

[20]  Fred A. Johnson,et al.  BAYESIAN INFERENCE AND DECISION THEORY—A FRAMEWORK FOR DECISION MAKING IN NATURAL RESOURCE MANAGEMENT , 2003 .

[21]  Andrew J. Tyre,et al.  A simple method for dealing with large state spaces , 2012 .

[22]  Hugh P Possingham,et al.  Optimal adaptive management for the translocation of a threatened species. , 2009, Ecological applications : a publication of the Ecological Society of America.

[23]  Marc Mangel,et al.  Dynamic models in behavioural and evolutionary ecology , 1988, Nature.

[24]  I. Chades,et al.  General rules for managing and surveying networks of pests, diseases, and endangered species , 2011, Proceedings of the National Academy of Sciences.

[25]  Paul D. Spencer,et al.  Optimal harvesting of fish populations with nonlinear rates of predation and autocorrelated environmental variability , 1997 .

[26]  Richard B. Norgaard,et al.  Sustainability and discounting the future , 1991 .

[27]  S. Creel,et al.  Meta-Analysis of Relationships between Human Offtake, Total Mortality and Population Dynamics of Gray Wolves (Canis lupus) , 2010, PloS one.

[28]  R. Bellman.  Dynamic programming. , 1957, Science.

[29]  Helen M. Regan,et al.  A TAXONOMY AND TREATMENT OF UNCERTAINTY FOR ECOLOGY AND CONSERVATION BIOLOGY , 2002 .

[30]  W. Kendall,et al.  Optimal control of native predators , 2010 .

[31]  Thomas E. Nygren,et al.  Influence of positive affect on the subjective utility of gains and losses: it is just not worth the risk. , 1988, Journal of personality and social psychology.

[32]  Dennis L. Murray,et al.  Death from anthropogenic causes is partially compensatory in recovering wolf populations. , 2010 .

[33]  B. Williams.  Adaptive management of natural resources--framework and issues. , 2011, Journal of environmental management.

[34]  Michael J. Conroy,et al.  Application of decision theory to conservation management: recovery of Hector's dolphin , 2008 .

[35]  François Charpillet,et al.  Dynamic optimization over infinite-time horizon: web-building strategy in an orb-weaving spider as a case study. , 2006, Journal of theoretical biology.

[36]  J. Nichols,et al.  Monitoring for conservation. , 2006, Trends in ecology & evolution.

[37]  C. Walters,et al.  Adaptive Control of Fishing Systems , 1976 .

[38]  Hugh P. Possingham,et al.  Optimizing search strategies for invasive pests: learn before you leap , 2011 .

[39]  Byron K. Williams,et al.  Markov decision processes in natural resources management: Observability and uncertainty , 2009 .

[40]  Hugh P. Possingham,et al.  Optimal release strategies for biological control agents: an application of stochastic dynamic programming to population management , 2000 .

[41]  R. Lande.  Risks of Population Extinction from Demographic and Environmental Stochasticity and Random Catastrophes , 1993, The American Naturalist.

[42]  Roger Pradel,et al.  Capture–recapture population growth rate as a robust tool against detection heterogeneity for population management , 2011 .

[43]  H. Raiffa,et al.  Applied Statistical Decision Theory. , 1961 .

[44]  Wayne M. Getz,et al.  The use of stochastic dynamic programming in optimal landscape reconstruction for metapopulations , 2003 .

[45]  E. J. Milner-Gulland,et al.  A STOCHASTIC DYNAMIC PROGRAMMING MODEL FOR THE MANAGEMENT OF THE SAIGA ANTELOPE , 1997 .

[46]  Michael C Runge,et al.  Structured decision making as a conceptual framework to identify thresholds for conservation and management. , 2009, Ecological applications : a publication of the Ecological Society of America.

[47]  SCOTT A. FIELD,et al.  OPTIMIZING ALLOCATION OF MONITORING EFFORT UNDER ECONOMIC AND OBSERVATIONAL CONSTRAINTS , 2005 .

[48]  Hilde Karine Wam,et al.  Economists, time to team up with the ecologists! , 2010 .

[49]  Eve McDonald-Madden,et al.  Optimal timing for managed relocation of species faced with climate change , 2011 .

[50]  C. Walters.  Optimal Harvest Strategies for Salmon in Relation to Environmental Variability and Uncertain Production Parameters , 1975 .

[51]  B. Wiman,et al.  The usefulness of stability concepts in forest management when coping with increasing climate uncertainties , 2007 .

[52]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[53]  Cindy E. Hauser,et al.  Optimal control of Atlantic population Canada geese , 2005 .

[54]  I. Chades,et al.  Simple rules to contain an invasive species with a complex life cycle and high dispersal capacity , 2012 .

[55]  Hugh P. Possingham,et al.  Optimal Conservation of Migratory Species , 2007, PloS one.

[56]  Michael D. Intriligator,et al.  Mathematical optimization and economic theory , 1971 .

[57]  Hugh P. Possingham,et al.  The business of biodiversity: Applying decision theory principles to nature conservation , 2001 .

[58]  C. Winkler,et al.  An Optimization Technique for the Budworm Forest-Pest Model , 1975 .

[59]  R. Hilborn,et al.  Fisheries stock assessment and decision analysis: the Bayesian approach , 1997, Reviews in Fish Biology and Fisheries.

[60]  Craig Boutilier,et al.  Stochastic dynamic programming with factored representations , 2000, Artif. Intell..

[61]  C. S. Holling.  Resilience and Stability of Ecological Systems , 1973 .

[62]  S. Nash,et al.  Linear and Nonlinear Programming , 1987 .

[63]  Cindy E. Hauser,et al.  How we value the future affects our desire to learn. , 2008, Ecological applications : a publication of the Ecological Society of America.

[64]  Mark R. Lembersky,et al.  Optimal Policies for Managed Stands: An Infinite Horizon Markov Decision Process Approach , 1975 .

[65]  Katriona Shea,et al.  An integrated approach to management in epidemiology and pest control , 2000 .

[66]  Lynne Caughlan,et al.  Cost considerations for long-term ecological monitoring , 2001 .

[67]  Carl J. Walters,et al.  Adaptive Management of Renewable Resources , 1986 .

[68]  Michael C. Runge,et al.  An Introduction to Adaptive Management for Threatened and Endangered Species , 2011 .

[69]  Carl J. Walters,et al.  ECOLOGICAL OPTIMIZATION AND ADAPTIVE MANAGEMENT , 1978 .

[70]  James D. Nichols,et al.  Adaptive harvest management of North American waterfowl populations: a brief history and future prospects , 2007, Journal of Ornithology.

[71]  I. Chades,et al.  Setting Realistic Recovery Targets for Two Interacting Endangered Species, Sea Otter and Northern Abalone , 2012, Conservation biology : the journal of the Society for Conservation Biology.

[72]  Ronald A. Howard,et al.  Dynamic Programming and Markov Processes , 1960 .