Ensemble Bayesian model averaging using Markov Chain Monte Carlo sampling

Bayesian model averaging (BMA) has recently been proposed as a statistical method to calibrate forecast ensembles from numerical weather models. Successful implementation of BMA however, requires accurate estimates of the weights and variances of the individual competing models in the ensemble. In their seminal paper (Raftery et al. Mon Weather Rev 133:1155–1174, 2005) has recommended the Expectation–Maximization (EM) algorithm for BMA model training, even though global convergence of this algorithm cannot be guaranteed. In this paper, we compare the performance of the EM algorithm and the recently developed DiffeRential Evolution Adaptive Metropolis (DREAM) Markov Chain Monte Carlo (MCMC) algorithm for estimating the BMA weights and variances. Simulation experiments using 48-hour ensemble data of surface temperature and multi-model streamflow forecasts show that both methods produce similar results, and that their performance is unaffected by the length of the training data set. However, MCMC simulation with DREAM is capable of efficiently handling a wide variety of BMA predictive distributions, and provides useful information about the uncertainty associated with the estimated BMA weights and variances.

[1]  Dong-Jun Seo,et al.  Towards the characterization of streamflow simulation uncertainty through multimodel ensembles , 2004 .

[2]  Cajo J. F. ter Braak,et al.  Treatment of input uncertainty in hydrologic modeling: Doing hydrology backward with Markov chain Monte Carlo simulation , 2008 .

[3]  Balaji Rajagopalan,et al.  Categorical Climate Forecasts through Regularization and Optimal Combination of Multiple GCM Ensembles , 2002 .

[4]  O. Papasouliotis,et al.  Long‐range, critical‐point dynamics in oil field flow rate data , 2006 .

[5]  Jasper A. Vrugt,et al.  Combining multiobjective optimization and Bayesian model averaging to calibrate forecast ensembles of soil hydraulic models , 2008 .

[6]  D. Higdon,et al.  Accelerating Markov Chain Monte Carlo Simulation by Differential Evolution with Self-Adaptive Randomized Subspace Sampling , 2009 .

[7]  Ming Ye,et al.  Maximum likelihood Bayesian averaging of spatial variability models in unsaturated fractured tuff , 2003 .

[8]  S. Sorooshian,et al.  A Shuffled Complex Evolution Metropolis algorithm for optimization and uncertainty assessment of hydrologic model parameters , 2002 .

[9]  Andrew P. Morse,et al.  DEVELOPMENT OF A EUROPEAN MULTIMODEL ENSEMBLE SYSTEM FOR SEASONAL-TO-INTERANNUAL PREDICTION (DEMETER) , 2004 .

[10]  David S. Richardson,et al.  Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size , 2001 .

[11]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[12]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[13]  A. Hense,et al.  A Bayesian approach to climate model evaluation and multi‐model averaging with an application to global mean surface temperatures from IPCC AR4 coupled climate models , 2006 .

[14]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[15]  T. N. Krishnamurti,et al.  Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble. , 1999, Science.

[16]  A. Barnston,et al.  Multimodel Ensembling in Seasonal Climate Forecasting at IRI , 2003 .

[17]  Qingyun Duan,et al.  An integrated hydrologic Bayesian multimodel combination framework: Confronting input, parameter, and model structural uncertainty in hydrologic prediction , 2006 .

[18]  E. Grimit,et al.  Initial Results of a Mesoscale Short-Range Ensemble Forecasting System over the Pacific Northwest , 2002 .

[19]  Thomas M. Hamill.  Comments on “Calibrated Surface Temperature Forecasts from the Canadian Ensemble Prediction System Using Bayesian Model Averaging” , 2007 .

[20]  F. Molteni,et al.  The ECMWF Ensemble Prediction System: Methodology and validation , 1996 .

[21]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[22]  Renate Hagedorn,et al.  The rationale behind the success of multi-model ensembles in seasonal forecasting-II , 2005 .

[23]  Bruce A. Robinson,et al.  Treatment of uncertainty using ensemble methods: Comparison of sequential data assimilation and Bayesian model averaging , 2007 .

[24]  S. Sorooshian,et al.  Effective and efficient global optimization for conceptual rainfall‐runoff models , 1992 .

[25]  D. Madigan,et al.  Bayesian Model Averaging for Linear Regression Models , 1997 .

[26]  Martyn P. Clark,et al.  Multi‐objective calibration of forecast ensembles using Bayesian model averaging , 2006 .

[27]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[28]  A. Raftery,et al.  Calibrated Surface Temperature Forecasts from the Canadian Ensemble Prediction System Using Bayesian Model Averaging , 2007 .

[29]  Thomas M. Hamill,et al.  Verification of Eta–RSM Short-Range Ensemble Forecasts , 1997 .

[30]  Adrian E. Raftery,et al.  Weather Forecasting with Ensemble Methods , 2005, Science.

[31]  J. M. Sloughter,et al.  Probabilistic Quantitative Precipitation Forecasting Using Bayesian Model Averaging , 2007 .

[32]  S. P. Neuman,et al.  Maximum likelihood Bayesian averaging of uncertain model predictions , 2003 .