Estimating correlated observation error statistics using an ensemble transform Kalman filter

For certain observing types, such as those that are remotely sensed, the observation errors are correlated and these correlations are state- and time-dependent. In this work, we develop a method for diagnosing and incorporating spatially correlated and time-dependent observation error in an ensemble data assimilation system. The method combines an ensemble transform Kalman filter with a method that uses statistical averages of background and analysis innovations to provide an estimate of the observation error covariance matrix. To evaluate the performance of the method, we perform identical twin experiments using the Lorenz '96 and Kuramoto-Sivashinsky models. Using our approach, a good approximation to the true observation error covariance can be recovered in cases where the initial estimate of the error covariance is incorrect. Spatial observation error covariances where the length scale of the true covariance changes slowly in time can also be captured. We find that using the estimated correlated observation error in the assimilation improves the analysis.

[1]  Lloyd N. Trefethen,et al.  Fourth-Order Time-Stepping for Stiff PDEs , 2005, SIAM J. Sci. Comput..

[2]  A. Lorenc,et al.  Operational implementation of a hybrid ensemble/4D‐Var global data assimilation system at the Met Office , 2013 .

[3]  Stephen E. Cohn,et al.  Treatment of Observation Error due to Unresolved Scales in Atmospheric Data Assimilation , 2006 .

[4]  Nancy Nichols,et al.  Correlated observation errors in data assimilation , 2008 .

[5]  Craig H. Bishop,et al.  Ensemble covariances adaptively localized with ECO-RAP. Part 1: tests on simple error models , 2009 .

[6]  Nancy Nichols,et al.  Unbiased ensemble square root filters , 2007 .

[7]  R. Daley.  Atmospheric Data Analysis , 1991 .

[8]  J. Whitaker,et al.  Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter , 2001 .

[9]  Bartosz Protas,et al.  Adjoint-based optimization of PDE systems with alternative gradients , 2008, J. Comput. Phys..

[10]  S. Cox,et al.  Exponential Time Differencing for Stiff Systems , 2002 .

[11]  P. Bickel,et al.  Regularized estimation of large covariance matrices , 2008, 0803.1909.

[12]  Craig H. Bishop,et al.  Adaptive sampling with the ensemble transform Kalman filter , 2001 .

[13]  P. L. Houtekamer,et al.  Ensemble Kalman filtering , 2005 .

[14]  E. Lorenz.  Predictability of Weather and Climate: Predictability – a problem partly solved , 2006 .

[15]  B. Hunt,et al.  A comparative study of 4D-VAR and a 4D Ensemble Kalman Filter: perfect model simulations with Lorenz-96 , 2007 .

[16]  Takemasa Miyoshi,et al.  Ensemble Kalman Filter and 4D-Var Intercomparison with the Japanese Operational Global Analysis and Prediction System , 2010 .

[17]  Copenhagen,et al.  AVERAGE PATTERNS OF SPATIOTEMPORAL CHAOS : A BOUNDARY EFFECT , 1998, chao-dyn/9805003.

[18]  Paul Poli,et al.  Diagnosis of observation, background and analysis‐error statistics in observation space , 2005 .

[19]  Niels Bormann,et al.  Estimates of spatial and interchannel observation‐error characteristics for current sounder radiances for numerical weather prediction. II: Application to AIRS and IASI data , 2010 .

[20]  Ross N. Bannister,et al.  A review of forecast error covariance statistics in atmospheric variational data assimilation. I: Characteristics and measurements of forecast error covariances , 2008 .

[21]  Gene H. Golub,et al.  Matrix computations , 1983 .

[22]  Preprint Mps,et al.  Representativity error for temperature and humidity using the Met Office high resolution model , 2012 .

[23]  Milija Zupanski,et al.  Comparison of sequential data assimilation methods for the Kuramoto–Sivashinsky equation , 2009 .

[24]  Nancy Nichols,et al.  Observation error correlations in IASI radiance data , 2009 .

[25]  Bartosz Protas,et al.  Regularization of the backward-in-time Kuramoto-Sivashinsky equation , 2010, J. Comput. Appl. Math..

[26]  W. Briggs.  Statistical Methods in the Atmospheric Sciences , 2007 .

[27]  Jeffrey L. Anderson.  An Ensemble Adjustment Kalman Filter for Data Assimilation , 2001 .

[28]  Istvan Szunyogh,et al.  A Local Ensemble Kalman Filter for Atmospheric Data Assimilation , 2002 .

[29]  J. Eyre,et al.  Accounting for correlated error in the assimilation of high‐resolution sounder data , 2014 .

[30]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[31]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[32]  Anthony Hollingsworth,et al.  The statistical structure of short-range forecast errors as determined from radiosonde data , 1986 .

[33]  A. Kanazawa,et al.  Analysis beyond Leading Twist for F2p-n Moments , 1980 .

[34]  E. Kalnay,et al.  Simultaneous estimation of covariance inflation and observation errors within an ensemble Kalman filter , 2009 .

[35]  Daniel Hodyss,et al.  Ensemble covariances adaptively localized with ECO-RAP. Part 2: a strategy for the atmosphere , 2009 .

[36]  R. Bannister.  A review of forecast error covariance statistics in atmospheric variational data assimilation. II: Modelling the forecast error covariance statistics , 2008 .

[37]  Takemasa Miyoshi,et al.  Estimating and including observation-error correlations in data assimilation , 2013 .

[38]  Niels Bormann,et al.  Estimates of spatial and interchannel observation‐error characteristics for current sounder radiances for numerical weather prediction. I: Methods and application to ATOVS data , 2010 .

[39]  T. Hamill.  Interpretation of Rank Histograms for Verifying Ensemble Forecasts , 2001 .

[40]  Nancy Nichols,et al.  Data assimilation with correlated observation errors: experiments with a 1-D shallow water model , 2013 .

[41]  S. B. Healy,et al.  Use of discrete Fourier transforms in the 1D‐Var retrieval problem , 2005 .

[42]  M. Buehner.  Error Statistics in Data Assimilation: Estimation and Modelling , 2010 .

[43]  J. Dormand,et al.  A family of embedded Runge-Kutta formulae , 1980 .

[44]  G. Sivashinsky.  Nonlinear analysis of hydrodynamic instability in laminar flames—I. Derivation of basic equations , 1977 .

[45]  Daniel Hodyss,et al.  Ensemble covariances adaptively localized with ECO-RAP. Part 1: tests on simple error models , 2009 .

[46]  Yoshiki Kuramoto,et al.  Diffusion-Induced Chaos in Reaction Systems , 1978 .

[47]  Xunqiang Yin,et al.  An ensemble adjustment Kalman filter study for Argo data , 2010 .

[48]  Jean-Noël Thépaut,et al.  The Spatial Structure of Observation Errors in Atmospheric Motion Vectors from Geostationary Satellite Data , 2003 .

[49]  Istvan Szunyogh,et al.  A local ensemble Kalman filter for atmospheric data assimilation , 2004 .

[50]  Nancy Nichols,et al.  Estimating interchannel observation‐error correlations for IASI radiance data in the Met Office system † , 2014 .

[51]  임규호,et al.  Optimal sites for supplementary weather observations , 2011 .

[52]  Thomas M. Hamill,et al.  Ensemble Data Assimilation with the NCEP Global Forecast System , 2008 .

[53]  J. Eyre,et al.  Assimilation of IASI at the Met Office and assessment of its impact through observing system experiments , 2009 .

[54]  H. J. Thiébaux.  Anisotropic Correlation Functions for Objective Analysis , 1976 .

[55]  M. Buehner,et al.  Intercomparison of Variational Data Assimilation and the Ensemble Kalman Filter for Global Deterministic NWP. Part II: One-Month Experiments with Real Observations , 2010 .