Performance and reliability of multimodel hydrological ensemble simulations based on seventeen lumped models and a thousand catchments

Abstract. This work investigates the added value of ensembles constructed from seventeen lumped hydrological models against their simple average counterparts. It is thus hypothesized that there is more information provided by all the outputs of these models than by their single aggregated predictors. For all available 1061 catchments, results showed that the mean continuous ranked probability score of the ensemble simulations were better than the mean average error of the aggregated simulations, confirming the added value of retaining all the components of the model outputs. Reliability of the simulation ensembles is also achieved for about 30% of the catchments, as assessed by rank histograms and reliability plots. Nonetheless this imperfection, the ensemble simulations were shown to have better skills than the deterministic simulations at discriminating between events and non-events, as confirmed by relative operating characteristic scores especially for larger streamflows. From 7 to 10 models are deemed sufficient to construct ensembles with improved performance, based on a genetic algorithm search optimizing the continuous ranked probability score. In fact, many model subsets were found improving the performance of the reference ensemble. This is thus not essential to implement as much as seventeen lumped hydrological models. The gain in performance of the optimized subsets is accompanied by some improvement of the ensemble reliability in most cases. Nonetheless, a calibration of the predictive distribution is still needed for many catchments.

[1]  Ashish Sharma,et al.  Hydrological model selection: A Bayesian alternative , 2005 .

[2]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[3]  François Anctil,et al.  Tools for the assessment of hydrological ensemble forecasts obtained by neural networks , 2009 .

[4]  Simon Jaun,et al.  A probabilistic view on the August 2005 floods in the upper Rhine catchment , 2008 .

[5]  W. W. Peterson,et al.  The theory of signal detectability , 1954, Trans. IRE Prof. Group Inf. Theory.

[6]  J. Parajka,et al.  Scale effects in conceptual hydrological modeling , 2009 .

[7]  C. Loumagne,et al.  Procédures de décision multimodèle pour une prévision des crues en temps réel: Application au bassin supérieur de la Garonne , 1995 .

[8]  R. T. Clarke,et al.  Adaptive hydrological forecasting—a review / Revue des méthodes de prévision hydrologique ajustables , 1981 .

[9]  S. Sorooshian,et al.  Multimodel Combination Techniques for Analysis of Hydrological Simulations: Application to Distributed Model Intercomparison Project Results , 2006 .

[10]  Naresh Devineni,et al.  Multimodel ensembles of streamflow forecasts: Role of predictor state in developing optimal combinations , 2007, Water resources research.

[11]  S. L. Markstrom,et al.  The modular modeling system (MMS) — The physical process modeling component of a database-centered decision support system for water and power management , 1996 .

[12]  M. Clark,et al.  Probabilistic Quantitative Precipitation Estimation in Complex Terrain , 2005 .

[13]  Kuolin Hsu,et al.  Uncertainty assessment of hydrologic model states and parameters: Sequential data assimilation using the particle filter , 2005 .

[14]  T. N. Krishnamurti,et al.  Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble. , 1999, Science.

[15]  C. Bishop,et al.  Improvement of ensemble reliability with a new dressing kernel , 2005 .

[16]  J. Schaake,et al.  Correcting Errors in Streamflow Forecast Ensemble Mean and Spread , 2008 .

[17]  O. Talagrand,et al.  Evaluation of probabilistic prediction systems for a scalar variable , 2005 .

[18]  Florence Habets,et al.  Analysis of Near-Surface Atmospheric Variables: Validation of the SAFRAN Analysis over France , 2008 .

[19]  Dong-Jun Seo,et al.  The distributed model intercomparison project (DMIP): Motivation and experiment design , 2004 .

[20]  A. Jakeman,et al.  Computation of the instantaneous unit hydrograph and identifiable component flows with application to two small upland catchments , 1990 .

[21]  Tempei Hashino,et al.  Distributions-oriented verification of probability forecasts for small data samples , 2003 .

[22]  Andrew P. Morse,et al.  DEVELOPMENT OF A EUROPEAN MULTIMODEL ENSEMBLE SYSTEM FOR SEASONAL-TO-INTERANNUAL PREDICTION (DEMETER) , 2004 .

[23]  F. Atger.  Verification of intense precipitation forecasts from single models and ensemble prediction systems , 2001 .

[24]  E. Hansen,et al.  NUMERICAL SIMULATION OF THE RAINFALL-RUNOFF PROCESS ON A DAILY BASIS , 1973 .

[25]  Martyn P. Clark,et al.  The Hydrological Ensemble Prediction Experiment , 2007 .

[26]  Hoshin Vijai Gupta,et al.  Regionalization of constraints on expected watershed response behavior for improved predictions in ungauged basins , 2007 .

[27]  R. L. Winkler,et al.  Scoring Rules for Continuous Probability Distributions , 1976 .

[28]  Lucy Marshall,et al.  Towards dynamic catchment modelling: a Bayesian hierarchical mixtures of experts framework , 2007 .

[29]  S. Bergström,et al.  DEVELOPMENT OF A CONCEPTUAL DETERMINISTIC RAINFALL-RUNOFF MODEL , 1973 .

[30]  Vazken Andréassian,et al.  How can rainfall‐runoff models handle intercatchment groundwater flows? Theoretical study based on 1040 French catchments , 2007 .

[31]  T. Hamill.  Interpretation of Rank Histograms for Verifying Ensemble Forecasts , 2001 .

[32]  E. Martin,et al.  A meteorological estimation of relevant parameters for snow models , 1993 .

[33]  George Kuczera,et al.  Critical evaluation of parameter consistency and predictive uncertainty in hydrological modeling: A case study using Bayesian total error analysis , 2009 .

[34]  Paulin Coulibaly,et al.  Improving Daily Reservoir Inflow Forecasts with Model Combination , 2005 .

[35]  M. Mu,et al.  A comparison study of the methods of conditional nonlinear optimal perturbations and singular vectors in ensemble prediction , 2009 .

[36]  Nicolas Le Moine,et al.  Le bassin versant de surface vu par le souterrain : une voie d'amélioration des performances et du réalisme des modèles pluie-débit? , 2008 .

[37]  Martyn P. Clark,et al.  HEPEX: The Hydrological Ensemble Prediction Experiment , 2007 .

[38]  M. Sugawara,et al.  Automatic calibration of the tank model / L'étalonnage automatique d'un modèle à cisterne , 1979 .

[39]  R. Garçon.  Modèle global pluie-débit pour la prévision et la prédétermination des crues , 1999 .

[40]  Ashish Sharma,et al.  Modeling the catchment via mixtures: Issues of model specification and validation , 2005 .

[41]  Keith Beven,et al.  The future of distributed models: model calibration and uncertainty prediction. , 1992 .

[42]  Neill E. Bowler,et al.  On the Reliability and Calibration of Ensemble Forecasts , 2009 .

[43]  R. Moore,et al.  A distribution function approach to rainfall runoff modeling , 1981 .

[44]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[45]  D. Stensrud,et al.  Reliable Probabilistic Quantitative Precipitation Forecasts from a Short-Range Ensemble Forecasting System , 2007 .

[46]  Andrew P. Morse,et al.  Application and validation of a seasonal ensemble prediction system using a dynamic malaria model. , 2010 .

[47]  Renate Hagedorn,et al.  The rationale behind the success of multi-model ensembles in seasonal forecasting — I. Basic concept , 2005 .

[48]  V. Singh,et al.  Application and testing of the simple rainfall-runoff model SIMHYD , 2002 .

[49]  A. Shamseldin,et al.  Methods for combining the outputs of different rainfall–runoff models , 1997 .

[50]  Leonard A. Smith,et al.  Combining dynamical and statistical ensembles , 2003 .

[51]  Vazken Andréassian,et al.  The exponential store: a correct formulation for rainfall—runoff modelling , 2003 .

[52]  Cajo J. F. ter Braak,et al.  Treatment of input uncertainty in hydrologic modeling: Doing hydrology backward with Markov chain Monte Carlo simulation , 2008 .

[53]  François Anctil,et al.  Impact of spatial aggregation of inputs and parameters on the efficiency of rainfall‐runoff models: A theoretical study using chimera watersheds , 2004 .

[54]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[55]  C. Perrin,et al.  Does a large number of parameters enhance model performance? Comparative assessment of common catchment model structures on 429 catchments , 2001 .

[56]  S. Sorooshian,et al.  Multi-model ensemble hydrologic prediction using Bayesian model averaging , 2007 .

[57]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[58]  Anthony J. Jakeman,et al.  Assessing the impact of land use change on hydrology by ensemble modelling(LUCHEM) II: ensemble combinations and predictions , 2009 .

[59]  Dong-Jun Seo,et al.  Towards the characterization of streamflow simulation uncertainty through multimodel ensembles , 2004 .

[60]  Jim Freer,et al.  Ensemble evaluation of hydrological model hypotheses , 2010 .

[61]  Florian Pappenberger,et al.  Ensemble forecasting using TIGGE for the July–September 2008 floods in the Upper Huai catchment: a case study , 2010 .

[62]  V. Fortin,et al.  Probabilistic forecasting from ensemble prediction systems: Improving upon the best‐member method by using a different weight and dressing kernel for each member , 2006 .

[63]  Martyn P. Clark,et al.  Framework for Understanding Structural Errors (FUSE): A modular framework to diagnose differences between hydrological models , 2008 .

[64]  Francisco J. Doblas-Reyes,et al.  Forecast assimilation: a unified framework for the combination of multi-model weather and climate predictions , 2005 .

[65]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[66]  François Anctil,et al.  Improvement of rainfall-runoff forecasts through mean areal rainfall optimization , 2006 .

[67]  Dominique Thiéry.  Utilisation d'un modèle global pour identifier sur un niveau piézométrique des influences multiples dues à diverses activités humaines. , 1982 .

[68]  F. Anctil,et al.  Which potential evapotranspiration input for a lumped rainfall-runoff model?. Part 2: Towards a simple and efficient potential evapotranspiration model for rainfall-runoff modelling , 2005 .

[69]  Thomas M. Hamill,et al.  Ensemble Reforecasting: Improving Medium-Range Forecast Skill Using Retrospective Forecasts , 2004 .

[70]  C. Perrin,et al.  Improvement of a parsimonious model for streamflow simulation , 2003 .

[71]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[72]  Jasper A. Vrugt,et al.  Combining multiobjective optimization and Bayesian model averaging to calibrate forecast ensembles of soil hydraulic models , 2008 .

[73]  J. Whitaker,et al.  REFORECASTS An Important Dataset for Improving Weather Predictions , 2006 .

[74]  Jim Freer,et al.  Towards a limits of acceptability approach to the calibration of hydrological models : Extending observation error , 2009 .

[75]  G. Evensen.  Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics , 1994 .

[76]  S. P. Neuman,et al.  Maximum likelihood Bayesian averaging of uncertain model predictions , 2003 .

[77]  Nilo Nascimento,et al.  GR3J: a daily watershed model with three free parameters , 1999 .

[78]  Bruce A. Robinson,et al.  Treatment of uncertainty using ensemble methods: Comparison of sequential data assimilation and Bayesian model averaging , 2007 .

[79]  Maria L. Rizzo,et al.  A new test for multivariate normality , 2005 .

[80]  Matthew D. Wilson,et al.  Tracking the uncertainty in flood alerts driven by grand ensemble weather predictions , 2009 .

[81]  Thibault Mathevet,et al.  Dynamic averaging of rainfall‐runoff model simulations from complementary model parameterizations , 2006 .