Two Approximate Dynamic Programming Algorithms for Managing Complete SIS Networks

Inspired by the problem of best managing the invasive mosquito Aedes albopictus across the 17 Torres Straits islands of Australia, we aim at solving a Markov decision process on large Susceptible-Infected-Susceptible (SIS) networks that are highly connected. While dynamic programming approaches can solve sequential decision-making problems on sparsely connected networks, these approaches are intractable for highly connected networks. Inspired by our case study, we focus on problems where the probability of nodes changing state is low and propose two approximate dynamic programming approaches. The first approach is a modified version of value iteration where only those future states that are similar to the current state are accounted for. The second approach models the state space as continuous instead of binary, with an on-line algorithm that takes advantage of Bellman's adapted equation. We evaluate the resulting policies through simulations and provide a priority order to manage the 17 infested Torres Strait islands. Both algorithms show promise, with the continuous state approach being able to scale up to high dimensionality (50 nodes). This work provides a successful example of how AI algorithms can be designed to tackle challenging computational sustainability problems.

[1]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[2]  R. Bellman.  Dynamic programming. , 1957, Science.

[3]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[4]  John N. Tsitsiklis,et al.  Neuro-dynamic programming: an overview , 1995, Proceedings of 1995 34th IEEE Conference on Decision and Control.

[5]  Leslie Pack Kaelbling,et al.  On the Complexity of Solving Markov Decision Problems , 1995, UAI.

[6]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[7]  Jesse Hoey,et al.  SPUDD: Stochastic Planning using Decision Diagrams , 1999, UAI.

[8]  Alessandro Vespignani,et al.  Epidemic spreading in scale-free networks. , 2000, Physical review letters.

[9]  R. May,et al.  How Viruses Spread Among Computers and People , 2001, Science.

[10]  Robert A. Beezer.  Review of: Discrete Mathematics by Lászlo Lovász, Józef Pelikán, and Katalin K. Vesztergombi , 2003 .

[11]  J. Koella,et al.  Shared Control of Epidemiological Traits in a Coevolutionary Model of Host‐Parasite Interactions , 2003, The American Naturalist.

[12]  Tim Hesterberg,et al.  Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control , 2004, Technometrics.

[13]  Scott Sanner,et al.  Affine Algebraic Decision Diagrams (AADDs) and their Application to Structured Probabilistic Inference , 2005, IJCAI.

[14]  P. Poupart.  Exploiting structure to efficiently solve large scale partially observable Markov decision processes , 2005 .

[15]  Nathalie Peyrard,et al.  Mean Field Approximation of the Policy Iteration Algorithm for Graph-Based Markov Decision Processes , 2006, ECAI.

[16]  S. Ritchie,et al.  Discovery of a Widespread Infestation of Aedes albopictus in the Torres Strait, Australia , 2006, Journal of the American Mosquito Control Association.

[17]  Régis Sabbadin,et al.  Approximate Linear-Programming Algorithms for Graph-Based Markov Decision Processes , 2006, ECAI.

[18]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[19]  G. Winskel.  What Is Discrete Mathematics , 2007 .

[20]  Warren B. Powell,et al.  Approximate Dynamic Programming - Solving the Curses of Dimensionality , 2007 .

[21]  H. Possingham,et al.  Incorporating the Effects of Socioeconomic Uncertainty into Priority Setting for Conservation Investment , 2007, Conservation biology : the journal of the Society for Conservation Biology.

[22]  Tracy M. Rout,et al.  Managing beyond the invader: manipulating disturbance of natives simplifies control efforts , 2008 .

[23]  I. Chades,et al.  Conservation decision-making in large state spaces , 2010 .

[24]  David B. Shmoys,et al.  Maximizing the Spread of Cascades Using Network Design , 2010, UAI.

[25]  Duncan Fyfe Gillies,et al.  Probabilistic Approaches to Estimating the Quality of Information in Military Sensor Networks , 2010, Comput. J..

[26]  Olivier Buffet,et al.  Markov Decision Processes in Artificial Intelligence , 2010 .

[27]  I. Chades,et al.  Beyond stochastic dynamic programming: a heuristic sampling method for optimizing conservation decisions in very large state spaces , 2011 .

[28]  Warren B. Powell,et al.  “Approximate dynamic programming: Solving the curses of dimensionality” by Warren B. Powell , 2007, Wiley Series in Probability and Statistics.

[29]  I. Chades,et al.  General rules for managing and surveying networks of pests, diseases, and endangered species , 2011, Proceedings of the National Academy of Sciences.

[30]  Ljusk Ola Eriksson,et al.  Management of the risk of wind damage in forestry: a graph-based Markov decision process approach , 2011, Ann. Oper. Res..

[31]  C. Scoglio,et al.  On the existence of a threshold for preventive behavioral responses to suppress epidemic spreading , 2012, Scientific Reports.

[32]  Olivier Buffet,et al.  Adaptive Management of Migratory Birds Under Sea Level Rise , 2013, IJCAI.

[33]  Shlomo Zilberstein,et al.  Planning Under Uncertainty Using Reduced Models: Revisiting Determinization , 2014, ICAPS.

[34]  Alan Fern,et al.  Dynamic Resource Allocation for Optimizing Population Diffusion , 2014, AISTATS.

[35]  Mykel J. Kochenderfer,et al.  Control of epidemics on graphs , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[36]  H. Possingham,et al.  Prioritizing eradication actions on islands: it's not all or nothing , 2016 .

[37]  Peter G. Fennell,et al.  Limitations of discrete-time approaches to continuous-time contagion dynamics , 2016, Physical review. E.

[38]  I. Chades,et al.  Finding the best management policy to eradicate invasive species from spatial ecological networks with simultaneous actions , 2017 .

[39]  Martin Péron,et al.  Selecting simultaneous actions of different durations to optimally manage an ecological network , 2017 .