Detecting Improvements in Forecast Correlation Skill: Statistical Testing and Power Analysis

AbstractThe skill of weather and climate forecast systems is often assessed by calculating the correlation coefficient between past forecasts and their verifying observations. Improvements in forecast skill can thus be quantified by correlation differences. The uncertainty in the correlation difference needs to be assessed to judge whether the observed difference constitutes a genuine improvement, or is compatible with random sampling variations. A widely used statistical test for correlation difference is known to be unsuitable, because it assumes that the competing forecasting systems are independent. In this paper, appropriate statistical methods are reviewed to assess correlation differences when the competing forecasting systems are strongly correlated with one another. The methods are used to compare correlation skill between seasonal temperature forecasts that differ in initialization scheme and model resolution. A simple power analysis framework is proposed to estimate the probability of correctly...

[1]  Leandro B. Díaz,et al.  The ability of a multi-model seasonal forecasting ensemble to forecast the frequency of warm, cold and wet extremes , 2015 .

[2]  Stefan Siegert,et al.  A Bayesian framework for verification and recalibration of ensemble forecasts: How uncertain is NAO predictability? , 2015, 1504.01933.

[3]  T. Palmer,et al.  Impact of hindcast length on estimates of seasonal climate predictability , 2015, Geophysical research letters.

[4]  F. Pappenberger,et al.  ERA-Interim/Land: a global land surface reanalysis data set , 2015 .

[5]  Michael K. Tippett,et al.  Comparing Forecast Skill , 2014 .

[6]  Claire E. Bulgin,et al.  Sea surface temperature datasets for climate applications from Phase 1 of the European Space Agency Climate Change Initiative (SST CCI) , 2014 .

[7]  F. Doblas-Reyes,et al.  Ensemble of sea ice initial conditions for interannual climate predictions , 2014, Climate Dynamics.

[8]  D. Lüthi,et al.  Physical constraints for temperature biases in climate models , 2013 .

[9]  Francisco J. Doblas-Reyes,et al.  Seasonal climate predictability and forecasting: status and prospects , 2013 .

[10]  M. Kimoto,et al.  Initialized near-term regional climate change prediction , 2013, Nature Communications.

[11]  Klaus Wyser,et al.  EC-Earth V2.2: description and validation of a new seamless earth system prediction model , 2012, Climate Dynamics.

[12]  F. Doblas-Reyes,et al.  Sensitivity of decadal predictions to the initial atmospheric and oceanic perturbations , 2012, Climate Dynamics.

[13]  I. Jolliffe,et al.  Forecast verification : a practitioner's guide in atmospheric science , 2011 .

[14]  J. Thepaut,et al.  The ERA‐Interim reanalysis: configuration and performance of the data assimilation system , 2011 .

[15]  Lingyun Wu,et al.  Land‐atmosphere coupling and summer climate variability over East Asia , 2011 .

[16]  Daniel S. Wilks,et al.  Sampling distributions of the Brier score and Brier skill score under serial dependence , 2010 .

[17]  L. Kornblueh,et al.  Advancing decadal-scale climate prediction in the North Atlantic sector , 2008, Nature.

[18]  Patrick Dattalo,et al.  Statistical Power Analysis , 2008 .

[19]  G. Zou.  Toward using confidence intervals to compare correlations. , 2007, Psychological methods.

[20]  I. Jolliffe.  Uncertainty and Inference for Verification Measures , 2007 .

[21]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[22]  D. Lawrence,et al.  Regions of Strong Coupling Between Soil Moisture and Precipitation , 2004, Science.

[23]  H. Storch,et al.  Statistical Analysis in Climate Research , 2000 .

[24]  Kevin E. Trenberth,et al.  The Definition of El Niño. , 1997 .

[25]  J. Hurrell.  Decadal Trends in the North Atlantic Oscillation: Regional Temperatures and Precipitation , 1995, Science.

[26]  A. H. Murphy,et al.  Skill Scores and Correlation Coefficients in Model Verification , 1989 .

[27]  J. Rodgers,et al.  Thirteen ways to look at the correlation coefficient , 1988 .

[28]  J. H. Steiger.  Tests for comparing elements of a correlation matrix. , 1980 .

[29]  E. J. Williams.  The Comparison of Regression Variables , 1959 .

[30]  C. Field.  Managing the risks of extreme events and disasters to advance climate change adaption , 2012 .

[31]  R. E. Livezey,et al.  Statistical Field Significance and its Determination by Monte Carlo Techniques , 1983 .