Nonlinear stability of the ensemble Kalman filter with adaptive covariance inflation

The Ensemble Kalman filter and Ensemble square root filters are data assimilation methods used to combine high dimensional nonlinear models with observed data. These methods have proved to be indispensable tools in science and engineering as they allow computationally cheap, low dimensional ensemble state approximation for extremely high dimensional turbulent forecast models. From a theoretical perspective, these methods are poorly understood, with the exception of a recently established but still incomplete nonlinear stability theory. Moreover, recent numerical and theoretical studies of catastrophic filter divergence have indicated that stability is a genuine mathematical concern and can not be taken for granted in implementation. In this article we propose a simple modification of ensemble based methods which resolves these stability issues entirely. The method involves a new type of adaptive covariance inflation, which comes with minimal additional cost. We develop a complete nonlinear stability theory for the adaptive method, yielding Lyapunov functions and geometric ergodicity under weak assumptions. We present numerical evidence which suggests the adaptive methods have improved accuracy over standard methods and completely eliminate catastrophic filter divergence. This enhanced stability allows for the use of extremely cheap, unstable forecast integrators, which would otherwise lead to widespread filter malfunction.

[1]  J. Mandel,et al.  On the convergence of the ensemble Kalman filter , 2009, Applications of mathematics.

[2]  Jeffrey L. Anderson,et al.  An adaptive covariance inflation error correction algorithm for ensemble filters , 2007 .

[3]  A. Majda,et al.  Catastrophic filter divergence in filtering nonlinear dissipative systems , 2010 .

[4]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[5]  Xin T. Tong,et al.  Nonlinear stability and ergodicity of ensemble based Kalman filters , 2015, 1507.08307.

[6]  A. Stuart,et al.  Well-posedness and accuracy of the ensemble Kalman filter in discrete and continuous time , 2013, 1310.3167.

[7]  Andrew J Majda,et al.  Concrete ensemble Kalman filters with rigorous catastrophic filter divergence , 2015, Proceedings of the National Academy of Sciences.

[8]  Eugenia Kalnay,et al.  Atmospheric Modeling, Data Assimilation and Predictability , 2002 .

[9]  Brian R. Hunt,et al.  Ensemble data assimilation for hyperbolic systems , 2013 .

[10]  Florian Nadel,et al.  Stochastic Processes And Filtering Theory , 2016 .

[11]  Andrew J. Majda,et al.  A mechanism for catastrophic filter divergence in data assimilation for sparse observation networks , 2013 .

[12]  Jeffrey L. Anderson.  An Ensemble Adjustment Kalman Filter for Data Assimilation , 2001 .

[13]  Craig H. Bishop,et al.  Adaptive sampling with the ensemble transform Kalman filter , 2001 .

[14]  Andrew M. Stuart,et al.  Long-Time Asymptotics of the Filtering Distribution for Partially Observed Chaotic Dynamical Systems , 2014, SIAM/ASA J. Uncertain. Quantification.

[15]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[16]  T. Sauer,et al.  Adaptive ensemble Kalman filtering of non-linear systems , 2013 .

[17]  Andrew J. Majda,et al.  Filtering Complex Turbulent Systems , 2012 .

[18]  John Harlim,et al.  Adaptive error covariances estimation methods for ensemble Kalman filters , 2014, J. Comput. Phys..

[19]  Jeffrey L. Anderson,et al.  A Monte Carlo Implementation of the Nonlinear Filtering Problem to Produce Ensemble Assimilations and Forecasts , 1999 .

[20]  Fuqing Zhang,et al.  An adaptive covariance relaxation method for ensemble data assimilation , 2015 .

[21]  T. Bengtsson,et al.  Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants , 2007 .

[22]  Ramon van Handel,et al.  Ergodicity and stability of the conditional distributions of nondegenerate markov chains , 2011, 1101.1822.

[23]  J. Whitaker,et al.  Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter , 2001 .

[24]  Oliver G. Ernst,et al.  Analysis of the Ensemble and Polynomial Chaos Kalman Filters in Bayesian Inverse Problems , 2015, SIAM/ASA J. Uncertain. Quantification.

[25]  Andrew M. Stuart,et al.  Geometric Ergodicity of Some Hypo-Elliptic Diffusions for Particle Motions , 2002 .

[26]  Andrew J. Majda,et al.  Information theory and stochastics for multiscale nonlinear systems , 2005 .

[27]  F. Gland,et al.  Large sample asymptotics for the ensemble Kalman filter , 2009 .

[28]  Jeffrey L. Anderson.  Spatially and temporally varying adaptive covariance inflation for ensemble filters , 2009 .

[29]  Andrew J. Majda,et al.  Nonlinear Dynamics and Statistical Theories for Basic Geophysical Flows , 2006 .