Testing competing precipitation forecasts accurately and efficiently: The spatial prediction comparison test

AbstractWhich model is best? Many challenges exist when testing competing forecast models, especially for those with high spatial resolution. Spatial correlation, double penalties, and small-scale errors are just a few such challenges. Many new methods have been developed in recent decades to tackle these issues. The spatial prediction comparison test (SPCT), which was developed for general spatial fields and applied to wind speed, is applied here to precipitation fields; which pose many unique challenges in that they are not normally distributed, are marked by numerous zero-valued grid points, and verification results are particularly sensitive to small-scale errors and double penalties. The SPCT yields a statistical test that solves one important issue for verifying forecasts spatially by accounting for spatial correlation. Important for precipitation forecasts is that the test requires no distributional assumptions, is easy to perform, and can be applied efficiently to either gridded or nongridded spat...

[1]  I. Zawadzki,et al.  Scale Dependence of the Predictability of Precipitation from Continental Radar Images. Part II: Probability Forecasts , 2004 .

[2]  Thomas M. Hamill,et al.  Hypothesis Tests for Evaluating Numerical Precipitation Forecasts , 1999 .

[3]  Xuguang Wang,et al.  Hierarchical Cluster Analysis of a Convection-Allowing Ensemble during the Hazardous Weather Testbed 2009 Spring Experiment. Part I: Development of the Object-Oriented Cluster Analysis Method for Precipitation Fields , 2011 .

[4]  Jason E. Nachamkin.  Application of the Composite Method to the Spatial Forecast Verification Methods Intercomparison Dataset , 2009 .

[5]  C. Willmott,et al.  Ambiguities inherent in sums-of-squares-based error statistics , 2009 .

[6]  Michael E. Baldwin.  Objective verification of high-resolution WRF forecasts during 2005 NSSL/SPC Spring Program , 2005 .

[7]  C. Frei,et al.  SAL—A Novel Quality Measure for the Verification of Quantitative Precipitation Forecasts , 2008 .

[8]  Felix Ament,et al.  Assessing the Benefits of Convection-Permitting Models by Neighborhood Verification: Examples from MAP D-PHASE , 2010 .

[9]  Finn Lindgren,et al.  An image warping approach to spatio-temporal modelling , 2005 .

[10]  Azriel Rosenfeld,et al.  Sequential Operations in Digital Picture Processing , 1966, JACM.

[11]  S. Sorooshian,et al.  Evaluation of satellite-retrieved extreme precipitation rates across the central United States , 2011 .

[12]  C. Keil,et al.  A Displacement and Amplitude Score Employing an Optical Flow Technique , 2009 .

[13]  Eric Gilleland,et al.  Application of Spatial Verification Methods to Idealized and NWP-Gridded Precipitation Forecasts , 2009 .

[14]  Eric Gilleland,et al.  Confidence Intervals for Forecast Verification , 2010 .

[15]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[16]  Manfred Dorninger,et al.  Quantifying verification uncertainty by reference data variation , 2012 .

[17]  Barbara G. Brown,et al.  Forecast verification: current status and future directions , 2008 .

[18]  J. Louis,et al.  Distortion Representation of Forecast Errors , 1995 .

[19]  Yao Liang,et al.  Analysis of Spatial Similarities Between NEXRAD and NLDAS Precipitation Data Products , 2010, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[20]  F. Atger.  Verification of intense precipitation forecasts from single models and ensemble prediction systems , 2001 .

[21]  Eric Gilleland,et al.  Intercomparison of Spatial Forecast Verification Methods , 2009 .

[22]  Neil I. Fox,et al.  An Object-Oriented Multiscale Verification Scheme , 2010 .

[23]  F. Mesinger.  Bias Adjusted Precipitation Threat Scores , 2008 .

[24]  Michael E. Baldwin,et al.  Field Significance Revisited: Spatial Bias Errors in Forecasts as Applied to the Eta Model , 2006 .

[25]  Elizabeth E. Ebert,et al.  Neighborhood Verification: A Strategy for Rewarding Close Forecasts , 2009 .

[26]  Yuqiong Liu,et al.  A wavelet-based approach to assessing timing errors in hydrologic predictions , 2011 .

[27]  Caren Marzban,et al.  Cluster Analysis for Object-Oriented Verification of Fields: A Variation , 2008 .

[28]  C. F. Sirmans,et al.  Nonstationary multivariate process modeling through spatially varying coregionalization , 2004 .

[29]  Gunilla Borgefors,et al.  Distance transformations in digital images , 1986, Comput. Vis. Graph. Image Process..

[30]  B. Brown,et al.  Object-Based Verification of Precipitation Forecasts. Part I: Methodology and Application to Mesoscale Rain Areas , 2006 .

[31]  B. Casati,et al.  New Developments of the Intensity-Scale Technique within the Spatial Verification Methods Intercomparison Project , 2010 .

[32]  Kenneth J. Westrick,et al.  Does Increasing Horizontal Resolution Produce More Skillful Forecasts , 2002 .

[33]  Jason E. Nachamkin,et al.  Mesoscale Verification Using Meteorological Composites , 2004 .

[34]  Caren Marzban,et al.  Three Spatial Verification Techniques: Cluster Analysis, Variogram, and Optical Flow , 2009 .

[35]  B. Brown,et al.  The Method for Object-Based Diagnostic Evaluation (MODE) Applied to Numerical Forecasts from the 2005 NSSL/SPC Spring Program , 2009 .

[36]  N. Roberts,et al.  Scale-Selective Verification of Rainfall Accumulations from High-Resolution Forecasts of Convective Events , 2008 .

[37]  R. E. Livezey,et al.  Statistical Field Significance and its Determination by Monte Carlo Techniques , 1983 .

[38]  Juan B. Valdés,et al.  Evaluation of mesoscale convective systems in South America using multiple satellite products and an object‐based approach , 2011 .

[39]  Tiziana Paccagnella,et al.  The COSMO-LEPS mesoscale ensemble system: validation of the methodology and verification , 2005 .

[40]  Arnold Tafferner,et al.  Development and propagation of severe thunderstorms in the Upper Danube catchment area: Towards an integrated nowcasting and forecasting system using real-time data and high-resolution simulations , 2008 .

[41]  C. Willmott,et al.  Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance , 2005 .

[42]  P. Nurmi,et al.  Overview of methods for the verification of quantitative precipitation forecasts , 2008 .

[43]  Sheng Chen,et al.  Spatial verification using a true metric , 2011 .

[44]  Kenji Matsuura,et al.  On the use of dimensioned measures of error to evaluate the performance of spatial interpolators , 2006, Int. J. Geogr. Inf. Sci..

[45]  Christopher Grassotti,et al.  Feature calibration and alignment to represent model forecast errors: Empirical regularization , 2003 .

[46]  Efi Foufoula-Georgiou,et al.  A new metric for comparing precipitation patterns with an application to ensemble forecasts , 2005 .

[47]  S. J. Weiss,et al.  Probabilistic Forecast Guidance for Severe Thunderstorms Based on the Identification of Extreme Phenomena in Convection-Allowing Model Forecasts , 2011 .

[48]  V. Ducrocq,et al.  Point and areal validation of forecast precipitation fields , 2006 .

[49]  Benjamin R. J. Schwedler,et al.  Diagnosing the Sensitivity of Binary Image Measures to Bias, Location, and Event Frequency within a Forecast Verification Framework , 2011 .

[50]  F. Diebold,et al.  Comparing Predictive Accuracy , 1994, Business Cycles.

[51]  D. Stephenson,et al.  A new intensity‐scale approach for the verification of spatial precipitation forecasts , 2004 .

[52]  Noel A Cressie,et al.  Statistics for Spatial Data. , 1992 .

[53]  Marion Mittermaier,et al.  Using an intensity‐scale technique to assess the added benefit of high‐resolution model precipitation forecasts , 2006 .

[54]  B. Brown,et al.  Forecasts of Spatial Fields , 2012 .

[55]  W. Gallus.  Application of Object-Based Verification Techniques to Ensemble Precipitation Forecasts , 2010 .

[56]  R. Hoffman,et al.  A Technique for Assimilating SSM/I Observations of Marine Atmospheric Storms: Tests with ECMWF Analyses , 1996 .

[57]  Y. Benjamini,et al.  False Discovery Rates for Spatial Signals , 2007 .

[58]  Richard A. Levine,et al.  Wavelets and Field Forecast Verification , 1997 .

[59]  V. Lakshmanan,et al.  A Gaussian Mixture Model Approach to Forecast Verification , 2010 .

[60]  Caren Marzban,et al.  An Object-Oriented Verification of Three NWP Model Formulations via Cluster Analysis: An Objective and a Subjective Analysis , 2008 .

[61]  Daniel S. Wilks,et al.  Resampling Hypothesis Tests for Autocorrelated Fields , 1997 .

[62]  Zbynek Sokol,et al.  A radar-based verification of precipitation forecast for local convective storms , 2007 .

[63]  Finn Lindgren,et al.  Analyzing the Image Warp Forecast Verification Method on Precipitation Fields from the ICP , 2010 .

[64]  Elizabeth E. Ebert,et al.  Toward Better Understanding of the Contiguous Rain Area (CRA) Method for Spatial Forecast Verification , 2009 .

[65]  H. Wernli,et al.  Verification of quantitative precipitation forecasts on short time-scales: A fuzzy approach to handle timing errors with SAL , 2011 .

[66]  B. Brown,et al.  Object-Based Verification of Precipitation Forecasts. Part II: Application to Convective Rain Systems , 2006 .

[67]  Amanda S. Hering,et al.  Comparing Spatial Predictions , 2011, Technometrics.

[68]  Adrian Baddeley,et al.  spatstat: An R Package for Analyzing Spatial Point Patterns , 2005 .

[69]  S. Weygandt.  Scale sensitivities in model precipitation skill scores during IHOP , 2004 .

[70]  Efi Foufoula-Georgiou,et al.  Space‐time rainfall organization and its role in validating quantitative precipitation forecasts , 2000 .

[71]  Yongkang Xue,et al.  Assessing the dynamic‐downscaling ability over South America using the intensity‐scale verification technique , 2011 .

[72]  I. Jolliffe.  Uncertainty and Inference for Verification Measures , 2007 .

[73]  K. Wapler,et al.  Comparative verification of different nowcasting systems to support optimisation of thunderstorm warnings , 2012 .

[74]  Y. Xue,et al.  Assessing the dynamic‐downscaling ability over South America using the intensity‐scale verification technique , 2011 .

[75]  Keith F. Brill.  A General Analytic Method for Assessing Sensitivity to Bias of Performance Measures for Dichotomous Forecasts , 2009 .

[76]  Yongtao Guan,et al.  Assessing Isotropy for Spatial Point Processes , 2006, Biometrics.

[77]  Caren Marzban,et al.  Optical Flow for Verification , 2010 .

[78]  A. Baddeley.  Errors in binary images and an $Lsp p$ version of the Hausdorff metric , 1992 .

[79]  Jason E. Nachamkin,et al.  Evaluation of Heavy Precipitation Forecasts Using Composite-Based Methods: A Distributions-Oriented Approach , 2005 .

[80]  E. Gilleland.  Spatial Forecast Verification: Baddeley’s Delta Metric Applied to the ICP Test Cases , 2011 .

[81]  Caren Marzban,et al.  Verification with Variograms , 2009 .

[82]  Heini Wernli,et al.  Spatial Forecast Verification Methods Intercomparison Project: Application of the SAL Technique , 2009 .

[83]  Eric Gilleland,et al.  Verifying Forecasts Spatially , 2010 .

[84]  C. Keil,et al.  A Displacement-Based Error Measure Applied in a Regional Ensemble Forecasting System , 2007 .

[85]  Marion Mittermaier,et al.  A long‐term assessment of precipitation forecast skill using the Fractions Skill Score , 2013 .

[86]  J. Otkin,et al.  Assimilation of Surface-Based Boundary Layer Profiler Observations during a Cool-Season Weather Event Using an Observing System Simulation Experiment. Part II: Forecast Assessment , 2011 .

[87]  Nigel Roberts,et al.  Intercomparison of Spatial Forecast Verification Methods: Identifying Skillful Spatial Scales Using the Fractions Skill Score , 2010 .

[88]  Elizabeth E. Ebert,et al.  Fuzzy verification of high‐resolution gridded forecasts: a review and proposed framework , 2008 .

[89]  J. Weinman,et al.  The Use of Digital Warping of Microwave Integrated Water Vapor Imagery to Improve Forecasts of Marine Extratropical Cyclones , 1998 .

[90]  Eric Gilleland,et al.  Computationally Efficient Spatial Forecast Verification Using Baddeley’s Delta Image Metric , 2008 .

[91]  V. M. Karyampudi,et al.  The Effect of Assimilating Rain Rates Derived from Satellites and Lightning on Forecasts of the 1993 Superstorm , 1999 .

[92]  Jordan G. Powers,et al.  A Description of the Advanced Research WRF Version 2 , 2005 .

[93]  Caren Marzban,et al.  Cluster Analysis for Verification of Precipitation Fields , 2006 .

[94]  U. Damrath,et al.  Probabilistic precipitation forecasts from a deterministic model: a pragmatic approach , 2005 .

[95]  J. McBride,et al.  Verification of precipitation in weather systems: determination of systematic errors , 2000 .

[96]  Noel A Cressie,et al.  Nonparametric hypothesis testing for a spatial signal , 2002, IEEE Workshop on Statistical Signal Processing, 2003.

[97]  Harold E. Brooks,et al.  Objective Limits on Forecasting Skill of Rare Events , 2013 .

[98]  Athanasios C. Micheas,et al.  Cell identification and verification of QPF ensembles using shape analysis techniques , 2007 .

[99]  Jason J. Levit,et al.  Multiscale Statistical Properties of a High-Resolution Precipitation Forecast , 2001 .

[100]  Moti Segal,et al.  Improving probabilistic ensemble forecasts of convection through the application of QPF-POP relationships , 2011 .

[101]  F. Mesinger,et al.  Applying a General Analytic Method for Assessing Bias Sensitivity to Bias-Adjusted Threat and Equitable Threat Scores , 2009 .

[102]  Valérie Ventura,et al.  Controlling the Proportion of Falsely Rejected Hypotheses when Conducting Multiple Tests with Climatological Data , 2004 .

[103]  S. J. Weiss,et al.  Some Practical Considerations Regarding Horizontal Resolution in the First Generation of Operational Convection-Allowing NWP , 2008 .