Implicit particle filtering for models with partial noise, and an application to geomagnetic data assimilation

Abstract. Implicit particle filtering is a sequential Monte Carlo method for data assimilation, designed to keep the number of particles manageable by focussing attention on regions of large probability. These regions are found by minimizing, for each particle, a scalar function F of the state variables. Some previous implementations of the implicit filter rely on finding the Hessians of these functions. The calculation of the Hessians can be cumbersome if the state dimension is large or if the underlying physics are such that derivatives of F are difficult to calculate, as happens in many geophysical applications, in particular in models with partial noise, i.e. with a singular state covariance matrix. Examples of models with partial noise include models where uncertain dynamic equations are supplemented by conservation laws with zero uncertainty, or with higher order (in time) stochastic partial differential equations (PDE) or with PDEs driven by spatially smooth noise processes. We make the implicit particle filter applicable to such situations by combining gradient descent minimization with random maps and show that the filter is efficient, accurate and reliable because it operates in a subspace of the state space. As an example, we consider a system of nonlinear stochastic PDEs that is of importance in geomagnetic data assimilation.

[1]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[2]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[3]  A.H. Haddad,et al.  Applied optimal estimation , 1976, Proceedings of the IEEE.

[4]  B. Parlett The Symmetric Eigenvalue Problem , 1981 .

[5]  R. Fletcher Practical Methods of Optimization , 1988 .

[6]  Lance M. Leslie,et al.  Tropical Cyclone Prediction Using a Barotropic Model Initialized by a Generalized Inverse Method , 1993 .

[7]  N. Gordon,et al.  Novel approach to nonlinear/non-Gaussian Bayesian state estimation , 1993 .

[8]  P. Courtier,et al.  A strategy for operational implementation of 4D‐Var, using an incremental approach , 1994 .

[9]  Michael Ghil,et al.  Advanced data assimilation in strongly nonlinear dynamical systems , 1994 .

[10]  Robert F. Stengel,et al.  Optimal Control and Estimation , 1994 .

[11]  Olivier Talagrand,et al.  Assimilation of Observations, an Introduction (gtSpecial IssueltData Assimilation in Meteology and Oceanography: Theory and Practice) , 1997 .

[12]  Philippe Courtier,et al.  Dual formulation of four‐dimensional variational assimilation , 1997 .

[13]  D. Zupanski A General Weak Constraint Applicable to Operational 4DVAR Data Assimilation Systems , 1997 .

[14]  Philippe Courtier,et al.  Unified Notation for Data Assimilation : Operational, Sequential and Variational , 1997 .

[15]  Jeffrey K. Uhlmann,et al.  New extension of the Kalman filter to nonlinear systems , 1997, Defense, Security, and Sensing.

[16]  P. Moral Measure-valued processes and interacting particle systems. Application to nonlinear filtering problems , 1998 .

[17]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[18]  Robert N. Miller,et al.  Data assimilation into nonlinear stochastic models , 1999 .

[19]  Simon J. Godsill,et al.  On sequential Monte Carlo sampling methods for Bayesian filtering , 2000, Stat. Comput..

[20]  I. Chueshov Gevrey regularity of random attractors for stochastic reaction-diffusion equations , 2000 .

[21]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[22]  Joel Ferziger,et al.  Higher Order Methods for Incompressible Fluid Flow: by Deville, Fischer and Mund, Cambridge University Press, 499 pp. , 2003 .

[23]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[24]  Pierre Del Moral,et al.  Feynman-Kac formulae , 2004 .

[25]  G. Lord,et al.  A numerical scheme for stochastic PDEs with Gevrey regularity , 2004 .

[26]  A. Chorin,et al.  Stochastic Tools in Mathematics and Science , 2005 .

[27]  Y. Trémolet Accounting for an imperfect model in 4D‐Var , 2006 .

[28]  D. Nychka Data Assimilation” , 2006 .

[29]  Alexandre Fournier,et al.  A case for variational geomagnetic data assimilation: insights from a one-dimensional, nonlinear, and sparsely observed MHD system , 2007, 0705.1777.

[30]  P. Courtier,et al.  Variational Assimilation of Meteorological Observations With the Adjoint Vorticity Equation. I: Theory , 2007 .

[31]  Robert N. Miller,et al.  Representer‐based variational data assimilation in a nonlinear model of nearshore circulation , 2007 .

[32]  W. Kuang,et al.  Data assimilation in a sparsely observed one-dimensional modeled MHD system , 2007 .

[33]  P. Courtier,et al.  Variational Assimilation of Meteorological Observations With the Adjoint Vorticity Equation. Ii: Numerical Results , 2007 .

[34]  P. Bickel,et al.  Obstacles to High-Dimensional Particle Filtering , 2008 .

[35]  P. Bickel,et al.  Sharp failure rates for the bootstrap particle filter in high dimensions , 2008, 0805.3287.

[36]  P. Kloeden,et al.  Overcoming the order barrier in the numerical approximation of stochastic partial differential equations with additive space–time noise , 2009, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[37]  Jonathan Weare,et al.  Particle filtering with path sampling and an application to a bimodal ocean current model , 2009, J. Comput. Phys..

[38]  Sigal Gottlieb,et al.  Spectral Methods , 2019, Numerical Methods for Diffusion Phenomena in Building Physics.

[39]  Peter Jan,et al.  Particle Filtering in Geophysical Systems , 2009 .

[40]  A. Chorin,et al.  Implicit sampling for particle filters , 2009, Proceedings of the National Academy of Sciences.

[41]  Matthias Morzfeld,et al.  Implicit particle filters for data assimilation , 2010, 1005.4002.

[42]  M. Bocquet,et al.  Beyond Gaussian Statistical Modeling in Geophysical Data Assimilation , 2010 .

[43]  Gauthier Hulot,et al.  An Introduction to Data Assimilation and Predictability in Geomagnetism , 2010 .

[44]  T. A. Zang,et al.  Spectral Methods: Fundamentals in Single Domains , 2010 .

[45]  P. Leeuwen,et al.  Nonlinear data assimilation in geosciences: an extremely efficient particle filter , 2010 .

[46]  J. Aubert,et al.  Inferring internal properties of Earth's core dynamics and their evolution from surface observations and a numerical geodynamo model , 2011 .

[47]  Matthias Morzfeld,et al.  A random map implementation of implicit filters , 2011, J. Comput. Phys..

[48]  A. Chorin,et al.  Implicit Particle Methods and Their Connection with Variational Data Assimilation , 2012, 1205.1830.