Increasing the Skill of Probabilistic Forecasts: Understanding Performance Improvements from Model-Error Representations

Four model-error schemes for probabilistic forecasts over the contiguous United States with the WRFARW mesoscale ensemble system are evaluated in regard to performance. Including a model-error representation leads to significant increases in forecast skill near the surface as measured by the Brier score. Combining multiple model-error schemes results in the best-performing ensemble systems, indicating that current model error is still too complex to be represented by a single scheme alone. To understand the reasons for the improved performance, it is examined whether model-error representations increase skill merely by increasing the reliability and reducing the bias—which could also be achieved by postprocessing—or if they have additional benefits. Removing the bias results overall in the largest skill improvement. Forecasts with model-error schemes continue to have better skill than without, indicating that their benefit goes beyond bias reduction. Decomposing theBrier scoreintoits components revealsthat, in addition tothe spread-sensitivereliability, the resolution component is significantly improved. This indicates that the benefits of including a model-error representation go beyond increasing reliability. This is further substantiated when all forecasts are calibrated to have similar spread. The calibrated ensembles with model-error schemes consistently outperform the calibrated control ensemble. Including a model-error representation remains beneficial even if the ensemble systems are calibrated and/ or debiased. This suggests that the merits of model-error representations go beyond increasing spread and removing the mean error and can account for certain aspects of structural model uncertainty.

[1]  Thomas M. Hamill,et al.  Ensemble Calibration of 500-hPa Geopotential Height and 850-hPa and 2-m Temperatures Using Reforecasts , 2007 .

[2]  Warren Tennant,et al.  Using a Stochastic Kinetic Energy Backscatter Scheme to Improve MOGREPS Probabilistic Forecast Skill , 2011 .

[3]  R. Buizza,et al.  A Comparison of the ECMWF, MSC, and NCEP Global Ensemble Prediction Systems , 2005 .

[4]  Carolyn A. Reynolds,et al.  Stochastic Nature of Physical Parameterizations in Ensemble Prediction: A Stochastic Convection Approach , 2008 .

[5]  T. Palmer,et al.  Addressing model uncertainty in seasonal and annual dynamical ensemble forecasts , 2009 .

[6]  F. Atger,et al.  Spatial and Interannual Variability of the Reliability of Ensemble-Based Probabilistic Forecasts: Consequences for Calibration , 2003 .

[7]  G. Shutts,et al.  Assessing parametrization uncertainty associated with horizontal resolution in numerical weather prediction models , 2014, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[8]  T. Hamill.  Interpretation of Rank Histograms for Verifying Ensemble Forecasts , 2001 .

[9]  R. Plant,et al.  A Stochastic Parameterization for Deep Convection Based on Equilibrium Statistics , 2008 .

[10]  G. Shutts.  A kinetic energy backscatter algorithm for use in ensemble prediction systems , 2005 .

[11]  David J. Stensrud,et al.  Using Initial Condition and Model Physics Perturbations in Short-Range Ensemble Simulations of Mesoscale Convective Systems , 2000 .

[12]  G. Powers,et al.  A Description of the Advanced Research WRF Version 3 , 2008 .

[13]  Chris Snyder,et al.  Model Uncertainty in a Mesoscale Ensemble Prediction System: Stochastic versus Multiphysics Representations , 2011 .

[14]  Antje Weisheimer,et al.  Assessment of representations of model uncertainty in monthly and seasonal forecast ensembles , 2011 .

[15]  Clifford F. Mass,et al.  Aspects of Effective Mesoscale, Short-Range Ensemble Forecasting , 2005 .

[16]  P. Courtier,et al.  Correlation modelling on the sphere using a generalized diffusion equation , 2001 .

[17]  P. L. Houtekamer,et al.  Toward Random Sampling of Model Error in the Canadian Ensemble Prediction System , 2010 .

[18]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[19]  T. Hamill,et al.  Evaluation of Eta-RSM Ensemble Probabilistic Precipitation Forecasts , 1998 .

[20]  Renate Hagedorn,et al.  Probabilistic Forecast Calibration Using ECMWF and GFS Ensemble Reforecasts. Part II: Precipitation , 2008 .

[21]  Neill E. Bowler,et al.  The MOGREPS short‐range ensemble prediction system , 2008 .

[22]  I. Jolliffe,et al.  Forecast verification : a practitioner's guide in atmospheric science , 2011 .

[23]  Renate Hagedorn,et al.  Comparing TIGGE multimodel forecasts with reforecast‐calibrated ECMWF ensemble forecasts , 2012 .

[24]  Thomas Jung,et al.  Systematic Model Error: The Impact of Increased Horizontal Resolution versus Improved Stochastic and Deterministic Parameterizations , 2012 .

[25]  Yuejian Zhu,et al.  Initial perturbations based on the ensemble transform (ET) technique in the NCEP global operational forecast system , 2008 .

[26]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[27]  Chris Snyder,et al.  The U.S. Air ForceWeather Agency’s mesoscale ensemble: scientific description and performance results , 2011 .

[28]  T. Palmer,et al.  Stochastic representation of model uncertainties in the ECMWF ensemble prediction system , 2007 .

[29]  Francis W. Zwiers,et al.  Improved Seasonal Probability Forecasts , 2003 .

[30]  Paul Poli,et al.  Diagnosis of observation, background and analysis‐error statistics in observation space , 2005 .

[31]  Leonard A. Smith,et al.  Evaluating Probabilistic Forecasts Using Information Theory , 2002 .

[32]  Clifford F. Mass,et al.  IFPS and the Future of the National Weather Service , 2003 .

[33]  Renate Hagedorn,et al.  The rationale behind the success of multi-model ensembles in seasonal forecasting-II , 2005 .

[34]  Guillem Candille,et al.  A Regional Ensemble Prediction System Based on Moist Targeted Singular Vectors and Stochastic Parameter Perturbations , 2008 .

[35]  Thomas M. Hamill,et al.  Ensemble Reforecasting: Improving Medium-Range Forecast Skill Using Retrospective Forecasts , 2004 .

[36]  Charles R. Sampson,et al.  Impact of Resolution and Design on the U.S. Navy Global Ensemble Performance in the Tropics , 2011 .

[37]  Martin Leutbecher,et al.  A Spectral Stochastic Kinetic Energy Backscatter Scheme and Its Impact on Flow-Dependent Predictability in the ECMWF Ensemble Prediction System , 2009 .

[38]  V. Fortin,et al.  Probabilistic forecasting from ensemble prediction systems: Improving upon the best‐member method by using a different weight and dressing kernel for each member , 2006 .

[39]  H. Storch.  On the Use of “Inflation” in Statistical Downscaling , 1999 .

[40]  C. Snyder,et al.  Linear and non-linear response to parameter variations in a mesoscale model , 2009 .

[41]  J Berner,et al.  Impact of a quasi-stochastic cellular automaton backscatter scheme on the systematic error and seasonal prediction skill of a global climate model , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[42]  Jeffrey L. Anderson,et al.  Representing forecast error in a convection-permitting ensemble system , 2014 .

[43]  David S. Richardson,et al.  Effects of observation errors on the statistics for ensemble spread and reliability , 2004 .

[44]  Renate Hagedorn,et al.  Probabilistic Forecast Calibration Using ECMWF and GFS Ensemble Reforecasts. Part I: Two-Meter Temperatures , 2008 .

[45]  J. Dudhia,et al.  A New Vertical Diffusion Package with an Explicit Treatment of Entrainment Processes , 2006 .

[46]  Laure Raynaud,et al.  Impact of Stochastic Physics in a Convection-Permitting Ensemble , 2012 .

[47]  Neill E. Bowler,et al.  The local ETKF and SKEB: Upgrades to the MOGREPS short‐range ensemble prediction system , 2009 .

[48]  A. H. Murphy.  A New Vector Partition of the Probability Score , 1973 .

[49]  C. Bishop,et al.  Improvement of ensemble reliability with a new dressing kernel , 2005 .

[50]  S. Julier,et al.  Which Is Better, an Ensemble of Positive–Negative Pairs or a Centered Spherical Simplex Ensemble? , 2004 .