Novel measures for summarizing high-resolution forecast performance

Abstract. In ascertaining the performance of a high-resolution gridded forecast against an analysis, called the verification set, on the same grid, care must be taken to account for the over-accumulation of small-scale errors and double penalties. It is also useful to consider both location errors and intensity errors. In the last 2 decades, many new methods have been proposed for analyzing these kinds of verification sets. Many of these new methods involve fairly complicated strategies that do not naturally summarize forecast performance succinctly. This paper presents two new spatial-alignment performance measures, G and Gβ . The former is applied without any requirement for user decisions, while the latter has a single user-chosen parameter, β , that takes on a value from zero to one, where one corresponds to a perfect match and zero corresponds to the user's notion of a worst case. Unlike any previously proposed distance-based measure, both handle the often-encountered case in which all values in one or both of the verification set are zero. Moreover, its value is consistent if only a few grid points are nonzero.

[1]  R. Steinacker,et al.  A Transparent Method for the Analysis and Quality Evaluation of Irregularly Distributed and Noisy Observational Data , 2000 .

[2]  Julian Koch,et al.  The SPAtial EFficiency metric (SPAEF): multiple-component evaluation of spatial patterns for optimization of hydrological models , 2018 .

[3]  B. Brown,et al.  The Method for Object-Based Diagnostic Evaluation (MODE) Applied to Numerical Forecasts from the 2005 NSSL/SPC Spring Program , 2009 .

[4]  David Malah,et al.  A study of edge detection algorithms , 1982, Comput. Graph. Image Process..

[5]  Dominique Brunet,et al.  A Generalized Distance Transform: Theory and Applications to Weather Analysis and Forecasting , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[6]  J. McBride,et al.  Verification of precipitation in weather systems: determination of systematic errors , 2000 .

[7]  B. Brown,et al.  Object-Based Verification of Precipitation Forecasts. Part I: Methodology and Application to Mesoscale Rain Areas , 2006 .

[8]  N. Cressie,et al.  Great expectations and even greater exceedances from spatially referenced data , 2020, Spatial Statistics.

[9]  E. Gilleland Spatial Forecast Verification: Baddeley’s Delta Metric Applied to the ICP Test Cases , 2011 .

[10]  B. Brown,et al.  Object-Based Verification of Precipitation Forecasts. Part II: Application to Convective Rain Systems , 2006 .

[11]  B. Brown,et al.  The set-up of the Mesoscale Verification Inter-Comparison over Complex Terrain (MesoVICT) Project , 2018 .

[12]  Zbynek Sokol,et al.  A radar-based verification of precipitation forecast for local convective storms , 2007 .

[13]  Finn Lindgren,et al.  Analyzing the Image Warp Forecast Verification Method on Precipitation Fields from the ICP , 2010 .

[14]  Wim H. Hesselink,et al.  A General Algorithm for Computing Distance Transforms in Linear Time , 2000, ISMM.

[15]  S. J. Weiss,et al.  Some Practical Considerations Regarding Horizontal Resolution in the First Generation of Operational Convection-Allowing NWP , 2008 .

[16]  Rodger B. Grayson,et al.  Quantitative comparison of spatial fields for hydrological model assessment––some promising approaches , 2005 .

[17]  J. Sheffield,et al.  Spatial validation of large‐scale land surface models against monthly land surface temperature patterns using innovative performance metrics , 2016 .

[18]  Elizabeth E. Ebert,et al.  Fuzzy verification of high‐resolution gridded forecasts: a review and proposed framework , 2008 .

[19]  C. Keil,et al.  A Displacement and Amplitude Score Employing an Optical Flow Technique , 2009 .

[20]  M. Dorninger,et al.  MesoVICT: Mesoscale Verification Inter-Comparison over Complex Terrain , 2013 .

[21]  N. Roberts,et al.  Scale-Selective Verification of Rainfall Accumulations from High-Resolution Forecasts of Convective Events , 2008 .

[22]  Marion Mittermaier,et al.  A Strategy for Verifying Near-Convection-Resolving Model Forecasts at Observing Sites , 2014 .

[23]  D. Stephenson,et al.  A new intensity‐scale approach for the verification of spatial precipitation forecasts , 2004 .

[24]  Efi Foufoula-Georgiou,et al.  A new metric for comparing precipitation patterns with an application to ensemble forecasts , 2005 .

[25]  Arnold Tafferner,et al.  Cb-TRAM: Tracking and monitoring severe convection from onset over rapid development to mature phase using multi-channel Meteosat-8 SEVIRI data , 2008 .

[26]  Kenneth J. Westrick,et al.  Does Increasing Horizontal Resolution Produce More Skillful Forecasts , 2002 .

[27]  P. Nurmi,et al.  Overview of methods for the verification of quantitative precipitation forecasts , 2008 .

[28]  C. Keil,et al.  A Displacement-Based Error Measure Applied in a Regional Ensemble Forecasting System , 2007 .

[29]  Eric Gilleland,et al.  A New Characterization within the Spatial Verification Framework for False Alarms, Misses, and Overall Patterns , 2017 .

[30]  Simon Stisen,et al.  Toward a true spatial model evaluation in distributed hydrological modeling: Kappa statistics, Fuzzy theory, and EOF‐analysis benchmarked by the human perception and evaluated against a modeling case study , 2015 .

[31]  Adrian Baddeley,et al.  spatstat: An R Package for Analyzing Spatial Point Patterns , 2005 .

[32]  Eric Gilleland,et al.  Intercomparison of Spatial Forecast Verification Methods , 2009 .

[33]  Michael Weniger,et al.  Spatial verification using wavelet transforms: a review , 2016, 1605.03395.

[34]  Jordan G. Powers,et al.  A Description of the Advanced Research WRF Version 2 , 2005 .

[35]  Caren Marzban,et al.  Optical Flow for Verification , 2010 .

[36]  Michael E. Baldwin Objective verification of high-resolution WRF forecasts during 2005 NSSL/SPC Spring Program , 2005 .

[37]  G. Powers,et al.  A Description of the Advanced Research WRF Version 3 , 2008 .

[38]  Eric Gilleland,et al.  A Novel Set of Geometric Verification Test Fields with Application to Distance Measures , 2020, Monthly Weather Review.

[39]  Eric Gilleland,et al.  Testing competing precipitation forecasts accurately and efficiently: The spatial prediction comparison test , 2013 .

[40]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[41]  Jian Zhang,et al.  Loss Function Approaches to Predict a Spatial Quantile and Its Exceedance Region , 2008, Technometrics.

[42]  B. Brown,et al.  Forecasts of Spatial Fields , 2012 .

[43]  Eric Gilleland,et al.  Application of Spatial Verification Methods to Idealized and NWP-Gridded Precipitation Forecasts , 2009 .

[44]  A. Baddeley An Error Metric for Binary Images , 1992 .

[45]  A. Baddeley Errors in binary images and an $Lsp p$ version of the Hausdorff metric , 1992 .