A probability and decision‐model analysis of PROVOST seasonal multi‐model ensemble integrations

A probabilistic analysis is made of seasonal ensemble integrations from the PROVOST project (PRediction Of climate Variations On Seasonal to interannual Time-scales), with emphasis on the Brier score and related Murphy decomposition, and the relative operating characteristic. To illustrate the significance of these results to potential users, results from the analysis of the relative operating characteristic are input to a simple decision model. The decision-model analysis is used to define a user-specific objective measure of the economic value of seasonal forecasts. The analysis is made for two simple meteorological forecast conditions or ‘events’, E, based on 850 hPa temperature. The ensemble integrations result from integrating four different models over the period 1979–93. For each model a set of 9-member ensembles is generated by running from consecutive analyses. Results from the Brier skill score analysis taken over all northern hemisphere grid points indicate that, whilst the skill of individual-model ensembles is only marginally higher than a probabilistic forecast of climatological frequencies, the multi-model ensemble is substantially more skilful than climatology. Both reliability and resolution are better for the multi-model ensemble than for the individual-model ensembles. This improvement arises both from the use of different models in the ensemble, and from the enhanced ensemble size obtained by combining individual-model ensembles; the latter reason was found to be the more important. Brier skill scores are higher for years in which there were moderate or strong El Nino Southern Oscillation (ENSO) events. Over Europe, only the multi-model ensembles showed skill over climatology. Similar conclusions are reached from an analysis of the relative operating characteristic. Results from the decision-model analysis show that the economic value of seasonal forecasts is strongly dependent on the cost, C, to the user of taking precautionary action against E, in relation to the potential loss, L, if precautionary action is not taken and E occurs. However, based on the multi-model ensemble data, the economic value can be as much as 50% of the value of a hypothetical perfect deterministic forecast. For the hemisphere as a whole, value is enhanced by restriction to ENSO years. It is shown that there is potential economic value in seasonal forecasts for European users. However, the impact of ENSO on economic value over Europe is mixed; value is enhanced by El Nino only for some potential users with specific C/L. The techniques developed are applicable to complex E for arbitrary regions. Hence these techniques are proposed as the basis of an objective probabilistic and decision-model evaluation of operational seasonal ensemble forecasts.