Optimization methods to solve adaptive management problems

Determining the best management actions is challenging when critical information is missing. However, urgency and limited resources require that decisions must be made despite this uncertainty. The best practice method for managing uncertain systems is adaptive management, or learning by doing. Adaptive management problems can be solved optimally using decision-theoretic methods; the challenge for these methods is to represent current and future knowledge using easy-to-optimize representations. Significant methodological advances have been made since the seminal adaptive management work was published in the 1980s, but despite recent advances, guidance for implementing these approaches has been piecemeal and study-specific. There is a need to collate and summarize new work. Here, we classify methods and update the literature with the latest optimal or near-optimal approaches for solving adaptive management problems. We review three mathematical concepts required to solve adaptive management problems: Markov decision processes, sufficient statistics, and Bayes’ theorem. We provide a decision tree to determine whether adaptive management is appropriate and then group adaptive management approaches based on whether they learn only from the past (passive) or anticipate future learning (active). We discuss the assumptions made when using existing models and provide solution algorithms for each approach. Finally, we propose new areas of development that could inspire future research. For a long time, limited by the efficiency of the solution methods, recent techniques to efficiently solve partially observable decision problems now allow us to solve more realistic adaptive management problems such as imperfect detection and non-stationarity in systems.

[1]  R. Fisher,et al.  On the Mathematical Foundations of Theoretical Statistics , 1922 .

[2]  H. Raiffa,et al.  Applied Statistical Decision Theory. , 1961 .

[3]  Karl Johan Åström,et al.  Optimal control of Markov processes with incomplete state information , 1965 .

[4]  C. Walters Optimal Harvest Strategies for Salmon in Relation to Environmental Variability and Uncertain Production Parameters , 1975 .

[5]  C. Walters,et al.  Adaptive Control of Fishing Systems , 1976 .

[6]  Carl J. Walters,et al.  ECOLOGICAL OPTIMIZATION AND ADAPTIVE MANAGEMENT , 1978 .

[7]  W. Silvert The Price of Knowledge: Fisheries Management as a Research Tool , 1978 .

[8]  Carl J. Walters,et al.  Adaptive Management of Stock–Recruitment Systems , 1981 .

[9]  C. Walters,et al.  Measurement Errors and Uncertainty in Parameter Estimates for Stock and Recruitment , 1981 .

[10]  C. Walters,et al.  Effects of Measurement Errors on the Assessment of Stock–Recruitment Relationships , 1981 .

[11]  G. Monahan State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms , 1982 .

[12]  C. Clark,et al.  Uncertainty, search, and information in fisheries , 1983 .

[13]  Carl J. Walters,et al.  Adaptive Management of Renewable Resources , 1986 .

[14]  C. Walters,et al.  ADAPTIVE MANAGEMENT OF HARVEST RATES IN THE PRESENCE OF A RISK AVERSE UTILITY FUNCTION , 1987 .

[15]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[16]  Marc Mangel,et al.  Dynamic models in behavioural and evolutionary ecology , 1988, Nature.

[17]  Karl Johan Åström,et al.  Adaptive Control , 1989, Embedded Digital Control with Microcontrollers.

[18]  William S. Lovejoy,et al.  Computationally Feasible Bounds for Partially Observed Markov Decision Processes , 1991, Oper. Res..

[19]  Anthony Charles,et al.  Uncertainty and Information in Fishery Management Models: A Bayesian Updating Algorithm , 1992 .

[20]  Craig Boutilier,et al.  Using Abstractions for Decision-Theoretic Planning with Time Constraints , 1994, AAAI.

[21]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[22]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[23]  S. Frederick,et al.  Choosing fisheries harvest policies : when does uncertainty matter? , 1995 .

[24]  Björn Wittenmark,et al.  Adaptive Dual Control Methods: An Overview , 1995 .

[25]  Leslie Pack Kaelbling,et al.  Learning Policies for Partially Observable Environments: Scaling Up , 1997, ICML.

[26]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[27]  F. Johnson,et al.  MANAGING NORTH AMERICAN WATERFOWL IN THE FACE OF UNCERTAINTY , 1995 .

[28]  F. Johnson,et al.  Uncertainty and the Adaptive Management of Waterfowl Harvests , 1996 .

[29]  C. T. Moore,et al.  Uncertainty and the management of mallard harvests , 1997 .

[30]  Robert Givan,et al.  Bounded Parameter Markov Decision Processes , 1997, ECP.

[31]  C. Walters Challenges in adaptive management of riparian and coastal ecosystems , 1997 .

[32]  Ronen I. Brafman,et al.  A Heuristic Variable Grid Solution Method for POMDPs , 1997, AAAI/IAAI.

[33]  A. Parma What can adaptive management do for our fish, forests, food, and biodiversity? , 1998 .

[34]  Craig Boutilier,et al.  Sequential Optimality and Coordination in Multiagent Systems , 1999, IJCAI.

[35]  N. Filatov,et al.  Survey of adaptive dual control methods , 2000 .

[36]  Neil Immerman,et al.  The Complexity of Decentralized Control of Markov Decision Processes , 2000, UAI.

[37]  Eric A. Hansen,et al.  An Improved Grid-Based Approximation Algorithm for POMDPs , 2001, IJCAI.

[38]  A. Gill,et al.  Using stochastic dynamic programming to determine optimal fire management for Banksia ornata , 2001 .

[39]  Blai Bonet,et al.  An epsilon-Optimal Grid-Based Algorithm for Partially Observable Markov Decision Processes , 2002, ICML.

[40]  Helen M. Regan,et al.  A TAXONOMY AND TREATMENT OF UNCERTAINTY FOR ECOLOGY AND CONSERVATION BIOLOGY , 2002 .

[41]  F. Johnson,et al.  Conditions and Limitations on Learning in the Adaptive Management of Mallard Harvests , 2002 .

[42]  François Charpillet,et al.  A heuristic approach for solving decentralized-POMDP: assessment on the pursuit problem , 2002, SAC '02.

[43]  Joelle Pineau,et al.  Point-based value iteration: An anytime algorithm for POMDPs , 2003, IJCAI.

[44]  Anne Condon,et al.  On the undecidability of probabilistic planning and related stochastic optimization problems , 2003, Artif. Intell..

[45]  W. Hanemann,et al.  Fishery Management Under Multiple Uncertainty , 2004 .

[46]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[47]  Nikos A. Vlassis,et al.  Perseus: Randomized Point-based Value Iteration for POMDPs , 2005, J. Artif. Intell. Res..

[48]  C. S. Holling Adaptive Environmental Assessment and Management , 2005 .

[49]  Matthias Ehrgott,et al.  Multicriteria Optimization , 2005 .

[50]  I. Chades,et al.  Solving Multiagent Markov Decision Processes : A Forest Management Example , 2005 .

[51]  Laurent El Ghaoui,et al.  Robust Control of Markov Decision Processes with Uncertain Transition Matrices , 2005, Oper. Res..

[52]  B. White An economic analysis of ecological monitoring , 2005 .

[53]  P. Poupart Exploiting structure to efficiently solve large scale partially observable Markov decision processes , 2005 .

[54]  Matthias Ehrgott,et al.  Multicriteria Optimization (2. ed.) , 2005 .

[55]  Hugh P. Possingham,et al.  Prioritizing global conservation efforts , 2006, Nature.

[56]  François Charpillet,et al.  Dynamic optimization over infinite-time horizon: web-building strategy in an orb-weaving spider as a case study. , 2006, Journal of theoretical biology.

[57]  R. Gregory,et al.  Deconstructing adaptive management: criteria for applications to environmental management. , 2006, Ecological applications : a publication of the Ecological Society of America.

[58]  C. T. Moore,et al.  Optimal Regeneration Planning for Old-Growth Forest: Addressing Scientific Uncertainty in Endangered Species Recovery through Adaptive Management , 2006 .

[59]  H. Possingham,et al.  Active Adaptive Management for Conservation , 2007, Conservation biology : the journal of the Society for Conservation Biology.

[60]  George E. Monahan,et al.  A Survey of Partially Observable Markov Decision Processes: Theory, Models, and Algorithms , 2007 .

[61]  Cindy E. Hauser,et al.  Experimental or precautionary? Adaptive management over a range of time horizons , 2007 .

[62]  M. McCarthy Bayesian Methods for Ecology: Frontmatter , 2007 .

[63]  M. McCarthy Bayesian Methods for Ecology: CASE STUDIES , 2007 .

[64]  Shefali V. Mehta,et al.  Optimal detection and control strategies for invasive species management , 2007 .

[65]  Tracy M. Rout,et al.  Managing beyond the invader: manipulating disturbance of natives simplifies control efforts , 2008 .

[66]  David Hsu,et al.  SARSOP: Efficient Point-Based POMDP Planning by Approximating Optimally Reachable Belief Spaces , 2008, Robotics: Science and Systems.

[67]  Cindy E. Hauser,et al.  How we value the future affects our desire to learn. , 2008, Ecological applications : a publication of the Ecological Society of America.

[68]  I. Chades,et al.  When to stop managing or surveying cryptic threatened species , 2008, Proceedings of the National Academy of Sciences.

[69]  Karl Johan Åström,et al.  Adaptive Control (2 rev. Dover ed.) , 2008 .

[70]  D. MacKenzie Getting the biggest bang for our conservation buck. , 2009, Trends in ecology & evolution.

[71]  Michael C Runge,et al.  Structured decision making as a conceptual framework to identify thresholds for conservation and management. , 2009, Ecological applications : a publication of the Ecological Society of America.

[72]  Carl D. Shapiro,et al.  Adaptive management: The U.S. Department of the Interior technical guide , 2009 .

[73]  Byron K. Williams,et al.  Markov decision processes in natural resources management: Observability and uncertainty , 2009 .

[74]  Hugh P Possingham,et al.  Optimal adaptive management for the translocation of a threatened species. , 2009, Ecological applications : a publication of the Ecological Society of America.

[75]  Michael C. Fu,et al.  Solving Continuous-State POMDPs via Density Projection , 2010, IEEE Transactions on Automatic Control.

[76]  Daniela Zhang Cziráková Subjektivita a semiabstraktné prvky v starej čínskej maľbe tušom , 2010 .

[77]  Cindy E. Hauser,et al.  Active adaptive conservation of threatened species in the face of uncertainty. , 2010, Ecological applications : a publication of the Ecological Society of America.

[78]  Hugh P Possingham,et al.  Should metapopulation restoration strategies increase patch area or number of patches? , 2010, Ecological applications : a publication of the Ecological Society of America.

[79]  Alana L. Moore,et al.  On Valuing Information in Adaptive‐Management Models , 2010, Conservation biology : the journal of the Society for Conservation Biology.

[80]  H. Possingham,et al.  Monitoring does not always count. , 2010, Trends in ecology & evolution.

[81]  David Hsu,et al.  Planning under Uncertainty for Robotic Tasks with Mixed Observability , 2010, Int. J. Robotics Res..

[82]  Olivier Buffet,et al.  Markov decision processes in artificial intelligence : MDPs, beyond MDPs and applications , 2010 .

[83]  Robert G. Haight,et al.  Optimal control of an invasive species with imperfect information about the level of infestation , 2010 .

[84]  Patrice Perny,et al.  On Finding Compromise Solutions in Multiobjective Markov Decision Processes , 2010, ECAI.

[85]  Olivier Buffet,et al.  Markov Decision Processes in Artificial Intelligence , 2010 .

[86]  C. T. Moore,et al.  An Adaptive Decision Framework for the Conservation of a Threatened Plant , 2011 .

[87]  Eve McDonald-Madden,et al.  Uncertainty and adaptive management for biodiversity conservation , 2011 .

[88]  B. Williams Resolving structural uncertainty in natural resources management using POMDP approaches , 2011 .

[89]  J. Nichols,et al.  Climate Change, Uncertainty, and Natural Resource Management , 2011 .

[90]  I. Chades,et al.  Allocating conservation resources between areas where persistence of a species is uncertain. , 2011, Ecological applications : a publication of the Ecological Society of America.

[91]  Michael C. Runge,et al.  An Introduction to Adaptive Management for Threatened and Endangered Species , 2011 .

[92]  I. Chades,et al.  General rules for managing and surveying networks of pests, diseases, and endangered species , 2011, Proceedings of the National Academy of Sciences.

[93]  David C. Smith,et al.  Human behaviour: the key source of uncertainty in fisheries management , 2011 .

[94]  Sarah J. Converse,et al.  Special Issue Article: Adaptive management for biodiversity conservation in an uncertain world Which uncertainty? Using expert elicitation and expected value of information to design an adaptive program , 2011 .

[95]  Byron K Williams,et al.  Passive and active adaptive management: approaches and an example. , 2011, Journal of environmental management.

[96]  I. Chades,et al.  Optimally managing under imperfect detection: A method for plant invasions , 2011 .

[97]  Byron K. Williams,et al.  Adaptive resource management and the value of information , 2011 .

[98]  Julien Martin,et al.  Structured decision making as a proactive approach to dealing with sea level rise in Florida , 2011 .

[99]  M. C. Runge,et al.  Adaptive Management of Reintroduction , 2012 .

[100]  Olivier Buffet,et al.  MOMDPs: A Solution for Modelling Adaptive Management Problems , 2012, AAAI.

[101]  I. Chades,et al.  Simple rules to contain an invasive species with a complex life cycle and high dispersal capacity , 2012 .

[102]  I. Chades,et al.  Which States Matter? An Application of an Intelligent Discretization Method to Solve a Continuous POMDP in Conservation Biology , 2012, PloS one.

[103]  James D. Nichols,et al.  Evaluating a multispecies adaptive management framework: Must uncertainty impede effective decision-making? , 2013 .

[104]  Michael R. Springborn,et al.  A density projection approach for non-trivial information dynamics: Adaptive management of stochastic natural resources , 2013 .

[105]  Daniel Kuhn,et al.  Robust Markov Decision Processes , 2013, Math. Oper. Res..

[106]  Michael C. Runge,et al.  Active adaptive management for reintroduction of an animal population , 2013 .

[107]  Olivier Buffet,et al.  Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence Optimally Solving Dec-POMDPs as Continuous-State MDPs , 2022 .

[108]  Martin J. Westgate,et al.  Adaptive management of biological systems: A review , 2013 .

[109]  Olivier Buffet,et al.  Adaptive Management of Migratory Birds Under Sea Level Rise , 2013, IJCAI.

[110]  Lucile Marescot,et al.  Complex decisions made simple: a primer on stochastic dynamic programming , 2013 .

[111]  Marie-Josée Cros,et al.  MDPtoolbox: a multi-platform toolbox to solve stochastic dynamic programming problems , 2014 .

[112]  C. Hunter,et al.  Optimal water depth management on river-fed National Wildlife Refuges in a changing climate , 2014, Climatic Change.

[113]  Krishna Pacifici,et al.  Addressing structural and observational uncertainty in resource management. , 2014, Journal of environmental management.

[114]  P. Kareiva,et al.  REVIEW: The evolving linkage between conservation science and practice at The Nature Conservancy , 2014, The Journal of applied ecology.

[115]  Robert G. Haight,et al.  Monitoring as a partially observable decision problem , 2014 .

[116]  Byron K Williams,et al.  Value of information in natural resource management: technical developments and application to pink-footed geese , 2015, Ecology and evolution.

[117]  I. Chades,et al.  Adapting environmental management to uncertain but inevitable change , 2015, Proceedings of the Royal Society B: Biological Sciences.

[118]  Shimon Whiteson,et al.  Point-Based Planning for Multi-Objective POMDPs , 2015, IJCAI.

[119]  E. Hernández-Andrade Optimal strategies for managing fetal growth restriction. , 2015, Minerva ginecologica.

[120]  Frans A. Oliehoek,et al.  Scalable Planning and Learning for Multiagent POMDPs , 2014, AAAI.

[121]  Thomas G. Dietterich,et al.  α-min: A Compact Approximate Solver For Finite-Horizon POMDPs , 2015, IJCAI.

[122]  Takayuki Osogami,et al.  Robust partially observable Markov decision process , 2015, ICML.

[123]  Gurutzeta Guillera-Arroita,et al.  When do we need more data? A primer on calculating the value of information for applied ecologists , 2015 .

[124]  I. Chades,et al.  Adaptive management for improving species conservation across the captive-wild spectrum , 2016, Biological Conservation.

[125]  Cindy E Hauser,et al.  Learning about colonization when managing metapopulations under an adaptive management framework. , 2016, Ecological applications : a publication of the Ecological Society of America.

[126]  Lynn A. Maguire,et al.  Timing of Protection of Critical Habitat Matters , 2017 .