Uncertainty Estimation with Deep Learning for Rainfall-Runoff Modelling

Deep Learning is becoming an increasingly important way to produce accurate hydrological predictions across a wide range of spatial and temporal scales. Uncertainty estimations are critical for actionable hydrological forecasting, and while standardized community benchmarks are becoming an increasingly important part of hydrological model development and research, similar tools for benchmarking uncertainty estimation are lacking. We establish an uncertainty estimation benchmarking procedure and present four Deep Learning baselines, out of which three are based on Mixture Density Networks and one is based on Monte Carlo dropout. Additionally, we provide a post-hoc model analysis to put forward some qualitative understanding of the resulting models. Most importantly however, we show that accurate, precise, and reliable uncertainty estimation can be achieved with Deep Learning.

[1]  Martyn P. Clark,et al.  Benchmarking and Process Diagnostics of Land Models , 2018, Journal of Hydrometeorology.

[2]  Simon King,et al.  Modelling the uncertainty in recovering articulation from acoustics , 2003, Comput. Speech Lang..

[3]  Jordi Vitrià,et al.  Modelling heterogeneous distributions with an Uncountable Mixture of Asymmetric Laplacians , 2019, NeurIPS.

[4]  D. McLaughlin,et al.  Hydrologic Data Assimilation with the Ensemble Kalman Filter , 2002 .

[5]  S. Hochreiter,et al.  Toward Improved Predictions in Ungauged Basins: Exploiting the Power of Machine Learning , 2019, Water Resources Research.

[6]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[7]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[8]  Sepp Hochreiter,et al.  Untersuchungen zu dynamischen neuronalen Netzen , 1991 .

[9]  Gabriel Abramowitz,et al.  Towards a benchmark for land surface models , 2005 .

[10]  S. Zhu,et al.  Internal and external coupling of Gaussian mixture model and deep recurrent network for probabilistic drought forecasting , 2020, International Journal of Environmental Science and Technology.

[11]  Simone Raudino,et al.  The Tyranny of Metrics , 2019, The European Legacy.

[12]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[13]  Andrew Binley,et al.  GLUE: 20 years on , 2014 .

[14]  R. Marty,et al.  A crash-testing framework for predictive uncertainty assessment when forecasting high flows in an extrapolation context , 2019, Hydrology and Earth System Sciences.

[15]  Balaji Lakshminarayanan,et al.  Deep Ensembles: A Loss Landscape Perspective , 2019, ArXiv.

[16]  Penny J. Johnes,et al.  Benchmarking the predictive capability of hydrological models for river flow and flood peak predictions across over 1000 catchments in Great Britain , 2019, Hydrology and Earth System Sciences.

[17]  Keith Beven,et al.  Facets of uncertainty: epistemic uncertainty, non-stationarity, likelihood, hypothesis testing, and communication , 2016 .

[18]  T. Lancaster,et al.  Bayesian Quantile Regression , 2005 .

[19]  Jim Tørresen,et al.  How do Mixture Density RNNs Predict the Future? , 2019, ArXiv.

[20]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[21]  S. Srihari.  Mixture Density Networks , 1994 .

[22]  Dimitri P. Solomatine,et al.  Data‐driven approaches for estimating uncertainty in rainfall‐runoff modelling , 2008 .

[23]  Douglas Eck,et al.  A Neural Representation of Sketch Drawings , 2017, ICLR.

[24]  S. Weijs,et al.  Why hydrological predictions should be evaluated using information theory , 2010 .

[25]  Keith Beven,et al.  So just why would a modeller choose to be incoherent , 2008 .

[26]  Jürgen Schmidhuber,et al.  Recurrent World Models Facilitate Policy Evolution , 2018, NeurIPS.

[27]  Chaopeng Shen,et al.  Enhancing Streamflow Forecast and Extracting Insights Using Long‐Short Term Memory Networks With Data Integration at Continental Scales , 2019, Water Resources Research.

[28]  Martyn P. Clark,et al.  Characterizing Uncertainty of the Hydrologic Impacts of Climate Change , 2016, Current Climate Change Reports.

[29]  Nikolay Laptev,et al.  Deep and Confident Prediction for Time Series at Uber , 2017, 2017 IEEE International Conference on Data Mining Workshops (ICDMW).

[30]  Milos Hauskrecht,et al.  Obtaining Well Calibrated Probabilities Using Bayesian Binning , 2015, AAAI.

[31]  Hoshin Vijai Gupta,et al.  The quantity and quality of information in hydrologic models , 2015 .

[32]  V. Klemeš,et al.  Operational Testing of Hydrological Simulation Models , 2022 .

[33]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[34]  Stefania Tamea,et al.  Verification tools for probabilistic forecasts of continuous hydrological variables , 2006 .

[35]  George Kuczera,et al.  Bayesian analysis of input uncertainty in hydrological modeling: 2. Application , 2006 .

[36]  Hoshin Vijai Gupta,et al.  A process‐based diagnostic approach to model evaluation: Application to the NWS distributed hydrologic model , 2008 .

[37]  Martyn P. Clark,et al.  Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance , 2014 .

[38]  Hoshin Vijai Gupta,et al.  Does Information Theory Provide a New Paradigm for Earth Science? Hypothesis Testing , 2020, Water Resources Research.

[39]  Daniel Kifer,et al.  Evaluating aleatoric and epistemic uncertainties of time series deep learning models for soil moisture predictions , 2019, ArXiv.

[40]  Sujay V. Kumar,et al.  Benchmarking NLDAS-2 Soil Moisture and Evapotranspiration to Separate Uncertainty Contributions. , 2016, Journal of hydrometeorology.

[41]  Thibault Mathevet,et al.  Hydrology and Earth System Sciences Hess Opinions " Crash Tests for a Standardized Evaluation of Hydrological Models " , 2022 .

[42]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[43]  R Govindaraju,et al.  ARTIFICIAL NEURAL NETWORKS IN HYDROLOGY: II, HYDROLOGIC APPLICATIONS , 2000 .

[44]  P. Dirmeyer,et al.  The Plumbing of Land Surface Models: Benchmarking Model Performance , 2015 .

[45]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[46]  Demetris Koutsoyiannis,et al.  A blueprint for process‐based modeling of uncertain hydrological systems , 2012 .

[47]  Yarin Gal,et al.  Understanding Measures of Uncertainty for Adversarial Example Detection , 2018, UAI.

[48]  James D. Brown,et al.  The Science of NOAA's Operational Hydrologic Ensemble Forecast Service , 2014 .

[49]  T. Cole.  Too many digits: the presentation of numerical data , 2015, Archives of Disease in Childhood.

[50]  Martyn P. Clark,et al.  The CAMELS data set: catchment attributes and meteorology for large-sample studies , 2017 .

[51]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[52]  Martyn P. Clark,et al.  Benchmarking of a Physically Based Hydrologic Model , 2017 .

[53]  Sepp Hochreiter,et al.  A note on leveraging synergy in multiple meteorological datasets with deep learning for rainfall-runoff modeling , 2020 .

[54]  Wei Gong,et al.  A review on statistical postprocessing methods for hydrometeorological ensemble forecasting , 2017 .

[55]  Fábio Ferreira,et al.  Conditional Density Estimation with Neural Networks: Best Practices and Benchmarks , 2019, ArXiv.

[56]  J. Kirchner.  Getting the right answers for the right reasons: Linking measurements, analyses, and models to advance the science of hydrology , 2006 .

[57]  George Kuczera,et al.  Understanding predictive uncertainty in hydrologic modeling: The challenge of identifying input and structural errors , 2010 .

[58]  Karsten Schulz,et al.  Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks , 2018, Hydrology and Earth System Sciences.

[59]  Yingchun Huang,et al.  The Applicability of LSTM-KNN Model for Real-Time Flood Forecasting in Different Climate Zones in China , 2020, Water.

[60]  Hoshin Vijai Gupta,et al.  Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling , 2009 .

[61]  NetworksChristopher,et al.  Mixture Density , 2008 .

[62]  Frederik Kratzert,et al.  What Role Does Hydrological Science Play in the Age of Machine Learning , 2020 .

[63]  Kuolin Hsu,et al.  Artificial Neural Network Modeling of the Rainfall‐Runoff Process , 1995 .

[64]  Keith Beven,et al.  A guide to good practice in modeling semantics for authors and referees , 2013 .