Stochastic and Robust Control of Water Resource Systems: Concepts, Methods and Applications

In order for water resources management to effectively cope with all the key drivers of global change (climate, demographic, economic, social, policy/law/institutional, and technology changes), it is essential that the traditional sector-by-sector management approach to water resources is transformed into a new paradigm, where water is considered as the principal and cross cutting medium for balancing food, energy security, and environmental sustainability. One major technical challenge in expanding the scope of water resources management across sectors and to the river basin level is to develop new methodologies and tools to cope with the increasing complexity of water systems. When dealing with large water resources systems, particularly with water reservoir networks, traditional non-linear, stochastic control approaches, like Stochastic Dynamic Programming (SDP), suffer from the curse of dimensionality that makes them essentially unusable. In this chapter we review the most advanced, general approaches available to overcome, or at least effectively mitigate, SDP limits in dealing with large water reservoir networks. Depending on the strategy adopted to alleviate the dimensionality burden we distinguish two classes of approaches: methods based on the restriction of the degrees of freedom of the control problem and methods based on the simplification of the water system model. Emphasis is given to the technical implications of the high dimension and highly non-linear nature of water reservoir systems and to the human dimension component involved in their operation. For each approach a real world numerical application is briefly presented.

[1]  Dimitri P. Solomatine,et al.  Neural networks and reinforcement learning in control of water systems , 2003 .

[2]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Stochastic Control , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[3]  Marcello Restelli,et al.  Tree‐based reinforcement learning for optimal water reservoir operation , 2010 .

[4]  Veena Sharma,et al.  Optimal multi-reservoir network control by two-phase neural network , 2004 .

[5]  Andrea Castelletti,et al.  A procedural approach to strengthening integration and participation in water resource planning , 2006, Environ. Model. Softw..

[6]  John N. Tsitsiklis,et al.  Feature-based methods for large scale dynamic programming , 2004, Machine Learning.

[7]  Ximing Cai,et al.  The role of hydrologic information in reservoir operation – Learning from historical releases , 2008 .

[8]  Alexander V. Lotov,et al.  Interactive Decision Maps: Approximation and Visualization of Pareto Frontier , 2004 .

[9]  David Q. Mayne,et al.  Constrained model predictive control: Stability and optimality , 2000, Autom..

[10]  Peter K. Kitanidis,et al.  Improved Dynamic Programming Methods for Optimal Control of Lumped-Parameter Stochastic Systems , 2001, Oper. Res..

[11]  Stuart E. Dreyfus,et al.  Applied Dynamic Programming , 1965 .

[12]  David Q. Mayne,et al.  Differential dynamic programming , 1972, The Mathematical Gazette.

[13]  Enrico Weber,et al.  SET-VALUED CONTROL LAWS IN TEV-DC CONTROL PROBLEMS , 2002 .

[14]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[15]  Harald Niederreiter,et al.  Random number generation and Quasi-Monte Carlo methods , 1992, CBMS-NSF regional conference series in applied mathematics.

[16]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  V. T. Chow,et al.  Discrete Differential Dynamic Programing Approach to Water Resources Systems Optimization , 1971 .

[19]  Augustine O. Esogbue,et al.  Dynamic Programming for Optimal Water Resource Systems Analysis , 1989 .

[20]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[21]  R. Soncini-Sessa,et al.  Remarks on the application of a risk-averse approach to the management of ‘El Carrizal’ reservoir , 1990 .

[22]  P. Kitanidis,et al.  Gradient dynamic programming for stochastic optimal control of multidimensional water resources systems , 1988 .

[23]  R. Soncini-Sessa,et al.  A Min-Max Approach to Reservoir Management , 1984 .

[24]  R. P. Oliveira,et al.  Operating rules for multireservoir systems , 1997 .

[25]  N. Buras,et al.  The dynamic programming approach to water‐resources development , 1961 .

[26]  Francesca Pianosi,et al.  Real‐time management of a multipurpose water reservoir with a heteroscedastic inflow model , 2009 .

[27]  Sh. Momtahen,et al.  Direct Search Approaches Using Genetic Algorithms for Optimization of Water Reservoir Operating Policies , 2007 .

[28]  Cristiano Cervellera,et al.  Deterministic design for neural network learning: an approach based on discrepancy , 2004, IEEE Transactions on Neural Networks.

[29]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[30]  Jin-Hee Lee,et al.  Stochastic optimization of multireservoir systems via reinforcement learning , 2007 .

[31]  André Turgeon A decomposition method for the long-term scheduling of reservoirs in series , 1981 .

[32]  Haralambos V. Vasiliadis,et al.  DEMAND – DRIVEN OPERATION OF RESERVOIRS USING UNCERTAINTY – BASED OPTIMAL OPERATING POLICIES , 1994 .

[33]  Ying Li,et al.  Numerical Solution of Continuous-State Dynamic Programs Using Linear and Spline Interpolation , 1993, Oper. Res..

[34]  Thomas E. Croley,et al.  Application of a distributed large basin runoff model in the Great Lakes basin , 2007 .

[35]  W. Trott,et al.  Optimization of Multiple Reservoir System , 1973 .

[36]  Andrea Castelletti,et al.  Integrated and Participatory Water Resources Management. Theory , 2007 .

[37]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[38]  R. Bellman Dynamic programming. , 1957, Science.

[39]  A. Turgeon,et al.  Learning disaggregation technique for the operation of long‐term hydroelectric power systems , 1994 .

[40]  Lyn C. Thomas,et al.  An aggregate stochastic dynamic programming model of multireservoir systems , 1997 .

[41]  Andrea Castelletti,et al.  Water reservoir control under economic, social and environmental constraints , 2008, Autom..

[42]  William W.-G. Yeh,et al.  Reservoir Management and Operations Models: A State‐of‐the‐Art Review , 1985 .

[43]  Andrea Castelletti,et al.  Topics on System Analysis and Integrated Water Resource Management , 2006 .

[44]  William S. Butcher,et al.  Optimization of the Operation of a Multiple‐Purpose Reservoir by Dynamic Programming , 1968 .

[45]  Rolf A. Deininger,et al.  Generalization of White's Method of Successive Approximations to Periodic Markovian Decision Processes , 1972, Oper. Res..

[46]  Warren B. Powell,et al.  Approximate Dynamic Programming - Solving the Curses of Dimensionality , 2007 .

[47]  Mark Thyer,et al.  Goulburn River experimental catchment data set , 2007 .

[48]  Carlo Piccardi,et al.  Infinite-horizon periodic minimax control problem , 1993 .

[49]  Kaisa Miettinen,et al.  Nonlinear multiobjective optimization , 1998, International series in operations research and management science.

[50]  Daniele de Rigo,et al.  Neuro-dynamic programming for designing water reservoir network management policies , 2007 .

[51]  Peter Stone,et al.  Batch reinforcement learning in a complex domain , 2007, AAMAS '07.

[52]  Cristiano Cervellera,et al.  Optimization of a large-scale water reservoir network by stochastic dynamic programming with efficient state space discretization , 2006, Eur. J. Oper. Res..

[53]  D. White,et al.  Dynamic programming, Markov chains, and the method of successive approximations , 1963 .

[54]  Miguel A. Mariño,et al.  RESERVOIR OPERATION BY ANT COLONY OPTIMIZATION ALGORITHMS , 2006 .

[55]  M. Sanguineti,et al.  Water reservoirs management under uncertainty by approximating networks and learning from data , 2007 .

[56]  R. Bellman,et al.  Polynomial approximation—a new computational technique in dynamic programming: Allocation processes , 1963 .

[57]  Richard M. Shane,et al.  TVA hydro scheduling model: theoretical aspects , 1981 .

[58]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[59]  Carlo Piccardi,et al.  Stochastic dynamic programming for reservoir optimal control: Dense discretization and inflow correlation assumption made possible by parallel computing , 1991 .

[60]  R. Soncini-Sessa,et al.  A decomposition approach to suboptimal control of discrete-time systems , 1994 .