Robustness and Accuracy of finite Ensemble Kalman filters in large dimensions

Contemporary data assimilation often involves more than a million prediction vari- ables. Finite ensemble Kalman filters (EnKF) have been developed by geoscientists. They are successful indispensable tools in science and engineering, because they allow for computationally cheap low ensemble state approximation for extremely large dimensional turbulent dynamical systems. The practical finite ensemble filter like EnKF necessarily involve modifications such as covariance inflation and localization, and it is a genuine mystery why they perform so well with small ensemble sizes in large dimensions. This paper provides the first rigorous stochastic analysis of the accuracy and covariance fidelity of finite EnKF in the practical regime where the ensemble size is much smaller than the large ambient dimension for finite EnKFs with random coefficients. A challenging issue overcome here is that finite EnKF in huge dimensions introduces unavoidable bias and model errors which need to be controlled and estimated.

[1]  Craig H. Bishop,et al.  Adaptive sampling with the ensemble transform Kalman filter , 2001 .

[2]  M. Rudelson,et al.  Hanson-Wright inequality and sub-gaussian concentration , 2013 .

[3]  J. Mandel,et al.  On the convergence of the ensemble Kalman filter , 2009, Applications of mathematics.

[4]  Pierre Del Moral,et al.  On the stability and the uniform propagation of chaos of Extended Ensemble Kalman-Bucy filters , 2016 .

[5]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[6]  Andrew J Majda,et al.  Concrete ensemble Kalman filters with rigorous catastrophic filter divergence , 2015, Proceedings of the National Academy of Sciences.

[7]  Jeffrey L. Anderson,et al.  An adaptive covariance inflation error correction algorithm for ensemble filters , 2007 .

[8]  Hamidou Tembine,et al.  Deterministic Mean-Field Ensemble Kalman Filtering , 2014, SIAM J. Sci. Comput..

[9]  Andrew J. Majda,et al.  Rigorous accuracy and robustness analysis for two-scale reduced random Kalman filters in high dimensions , 2016, 1606.09087.

[10]  A. Stuart,et al.  Well-posedness and accuracy of the ensemble Kalman filter in discrete and continuous time , 2013, 1310.3167.

[11]  Arnaud Doucet,et al.  A survey of convergence results on particle filtering methods for practitioners , 2002, IEEE Trans. Signal Process..

[12]  Pierre Del Moral,et al.  On the stability and the exponential concentration of Extended Kalman-Bucy filters , 2016 .

[13]  Andrew J. Majda,et al.  Filtering Complex Turbulent Systems , 2012 .

[14]  Xin T. Tong,et al.  Nonlinear stability and ergodicity of ensemble based Kalman filters , 2015, 1507.08307.

[15]  Tosio Kato.  A Short Introduction to Perturbation Theory for Linear Operators , 1982 .

[16]  Jeffrey L. Anderson.  Spatially and temporally varying adaptive covariance inflation for ensemble filters , 2009 .

[17]  Nancy Nichols,et al.  Unbiased ensemble square root filters , 2007 .

[18]  P. Bougerol.  Kalman filtering with random coefficients and contractions , 1993 .

[19]  Konrad Reif,et al.  Stochastic stability of the discrete-time extended Kalman filter , 1999, IEEE Trans. Autom. Control..

[20]  Pierre Del Moral,et al.  On the stability and the uniform propagation of chaos properties of Ensemble Kalman-Bucy filters , 2016, 1605.09329.

[21]  Andrew J. Majda,et al.  Filtering nonlinear dynamical systems with linear stochastic models , 2008 .

[22]  Christopher M. Danforth,et al.  Accounting for Model Errors in Ensemble Data Assimilation , 2009 .

[23]  Christopher K. Wikle,et al.  Atmospheric Modeling, Data Assimilation, and Predictability , 2005, Technometrics.

[24]  Andrew J Majda,et al.  Nonlinear stability of the ensemble Kalman filter with adaptive covariance inflation , 2015, 1507.08319.

[25]  T. Bengtsson,et al.  Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants , 2007 .

[26]  P. Moral.  Nonlinear filtering : Interacting particle resolution , 1997 .

[27]  Andrew J Majda,et al.  Elementary models for turbulent diffusion with complex physical features: eddy diffusivity, spectrum and intermittency , 2013, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[28]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[29]  A. Stuart,et al.  Analysis of the 3DVAR filter for the partially observed Lorenz'63 model , 2012, 1212.4923.

[30]  Jeffrey L. Anderson.  An Ensemble Adjustment Kalman Filter for Data Assimilation , 2001 .