On the use of a Huber norm for observation quality control in the ECMWF 4D‐Var

This article describes a number of important aspects that need to be considered when designing and implementing an observation quality control scheme in an NWP data assimilation system. It is shown how careful evaluation of innovation statistics provides valuable knowledge about the observation errors and help in the selection of a suitable observation error model. The focus of the article is on the statistical specification of the typical fat tails of the innovation distributions. In observation error specifications, like the one used previously at ECMWF, it is common to assume outliers to represent gross errors that are independent of the atmospheric state. The investigations in this article show that this is not a good assumption for almost all observing systems used in today's data assimilation systems. It is found that a Huber norm distribution is a very suitable distribution to describe most innovation statistics, after discarding systematically erroneous observations. The Huber norm is a robust method, making it safer to include outlier observations in the analysis step. Therefore the background quality control can safely be relaxed. The Huber norm has been implemented in the ECMWF assimilation system for in situ observations. The design, implementation and results from this implementation are described in this article. The general impact of using the Huber norm distribution is positive, compared to the previously used variational quality control method which gave virtually no weight to outliers. Case-studies show how the method improves the use of observations, especially for intense cyclones and other extreme events. It is also discussed how the Huber norm distribution can be used to identify systematic problems with observing systems.

[1]  P. Courtier,et al.  The ECMWF implementation of three‐dimensional variational assimilation (3D‐Var). II: Structure functions , 1998 .

[2]  P. Courtier,et al.  The ECMWF implementation of three‐dimensional variational assimilation (3D‐Var). I: Formulation , 1998 .

[3]  J. Thepaut,et al.  Assimilation and Modeling of the Atmospheric Hydrological Cycle in the ECMWF Forecasting System , 2005 .

[4]  P. Courtier,et al.  A strategy for operational implementation of 4D‐Var, using an incremental approach , 1994 .

[5]  Heikki Järvinen,et al.  Variational quality control , 1999 .

[6]  Heikki Järvinen,et al.  Variational assimilation of time sequences of surface observations with serially correlated errors , 1999 .

[7]  Arlindo da Silva,et al.  An adaptive buddy check for observational quality control , 2013 .

[8]  Stephen J. Wright,et al.  Data assimilation in weather forecasting: a case study in PDE-constrained optimization , 2009 .

[9]  Andrea Storto,et al.  Variational quality control of hydrographic profile data with non-Gaussian errors for global ocean variational data assimilation systems , 2016 .

[10]  Andrew C. Lorenc,et al.  Analysis methods for numerical weather prediction , 1986 .

[11]  A. Hollingsworth,et al.  Monitoring of observation and analysis quality by a data assimilation system , 1986 .

[12]  W. Symes,et al.  Robust inversion of seismic data using the Huber norm , 2003 .

[13]  N. B. Ingleby,et al.  Bayesian quality control using multivariate normal distributions , 1993 .

[14]  N. B. Ingleby,et al.  Treatment of Gross Errors Using Maximum Probability Theory , 1992 .

[15]  Peter J. Huber,et al.  John W. Tukey's contributions to robust statistics , 2002 .

[16]  D. F. Andrews,et al.  Robust Estimates of Location , 1972 .

[17]  J. Thepaut,et al.  The ERA‐Interim reanalysis: configuration and performance of the data assimilation system , 2011 .

[18]  Uwe Ulbrich,et al.  Three extreme storms over Europe in December 1999 , 2001 .

[19]  J. Thepaut,et al.  383 Assimilation and Modelling of the Hydrological Cycle : ECMWF ’ s Status and Plans , 2002 .

[20]  P. J. Huber.  The 1972 Wald Lecture Robust Statistics: A Review , 1972 .

[21]  Paul Poli,et al.  Diagnosis of observation, background and analysis‐error statistics in observation space , 2005 .

[22]  Andrew C. Lorenc,et al.  Objective quality control of observations using Bayesian methods. Theory, and a practical implementation , 1988 .

[23]  Massimo Bonavita,et al.  On the use of EDA background error variances in the ECMWF 4D‐Var , 2012 .