Observation Quality Control with a Robust Ensemble Kalman Filter

AbstractCurrent ensemble-based Kalman filter (EnKF) algorithms are not robust to gross observation errors caused by technical or human errors during the data collection process. In this paper, the authors consider two types of gross observational errors, additive statistical outliers and innovation outliers, and introduce a method to make EnKF robust to gross observation errors. Using both a one-dimensional linear system of dynamics and a 40-variable Lorenz model, the performance of the proposed robust ensemble Kalman filter (REnKF) was tested and it was found that the new approach greatly improves the performance of the filter in the presence of gross observation errors and leads to only a modest loss of accuracy with clean, outlier-free, observations.

[1]  N. B. Ingleby,et al.  Bayesian quality control using multivariate normal distributions , 1993 .

[2]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[3]  Nozer D. Singpurwalla,et al.  Understanding the Kalman Filter , 1983 .

[4]  Brian R. Hunt,et al.  A non‐Gaussian Ensemble Filter for Assimilating Infrequent Noisy Observations , 2007 .

[5]  Istvan Szunyogh,et al.  A local ensemble transform Kalman filter data assimilation system for the NCEP global model , 2008 .

[6]  N. F. Toda,et al.  Divergence in the Kalman Filter , 1967 .

[7]  Peter J. Huber,et al.  Robust Statistics , 2005, Wiley Series in Probability and Statistics.

[8]  Marc G. Genton,et al.  Discussion of 'Breakdown and Groups' , 2005 .

[9]  Rudolf Dutter,et al.  Robust time series analysis: a survey , 1987, Kybernetika.

[10]  M. Otto,et al.  Outliers in Time Series , 1972 .

[11]  L. Fahrmeir,et al.  Penalized likelihood smoothing in robust state space models , 1999 .

[12]  M. West.  Robust Sequential Approximate Bayesian Estimation , 1981 .

[13]  S.A. Kassam,et al.  Robust techniques for signal processing: A survey , 1985, Proceedings of the IEEE.

[14]  P. Houtekamer,et al.  Data Assimilation Using an Ensemble Kalman Filter Technique , 1998 .

[15]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[16]  M. West.  Outlier Models and Prior Distributions in Bayesian Linear Regression , 1984 .

[17]  Adrian E. Raftery,et al.  Non-Gaussian State-Space Modeling of Nonstationary Time Series: Comment: Robustness, Computation, and Non-Euclidean Models , 1987 .

[18]  Marc G. Genton,et al.  Breakdown-point for Spatially and Temporally Correlated Observations , 2003 .

[19]  M. Genton,et al.  A skewed Kalman filter , 2005 .

[20]  Marc G. Genton,et al.  Comprehensive definitions of breakdown points for independent and dependent observations , 2003 .

[21]  Laurent E. Calvet,et al.  Robust Filtering , 2012 .

[22]  Richard J. Meinhold,et al.  Robustification of Kalman Filter Models , 1989 .

[23]  P. Papantoni-Kazakos,et al.  OUTLIER RESISTANT PREDICTION FOR STATIONARY PROCESSES , 1994 .

[24]  J. Whitaker,et al.  Ensemble Data Assimilation without Perturbed Observations , 2002 .

[25]  Heikki Järvinen,et al.  Variational quality control , 1999 .

[26]  L. Fahrmeir,et al.  On kalman filtering, posterior mode estimation and fisher scoring in dynamic exponential family regression , 1991 .

[27]  K. Emanuel,et al.  Optimal Sites for Supplementary Weather Observations: Simulation with a Small Model , 1998 .

[28]  Peter Ruckdeschel,et al.  Optimally Robust Kalman Filtering , 2010 .

[29]  F. Hampel.  Contributions to the theory of robust estimation , 1968 .

[30]  Shen Jun,et al.  OPTIMAL ROBUST FILTERING , 1993 .

[31]  Adrian E. Raftery,et al.  Comment: Robustness, Computation, and Non-Euclidean Models , 1987 .

[32]  Ibrahim Hoteit,et al.  Robust Ensemble Filtering and Its Relation to Covariance Inflation in the Ensemble Kalman Filter , 2011, 1108.0158.

[33]  G. Evensen,et al.  Analysis Scheme in the Ensemble Kalman Filter , 1998 .

[34]  S. Mitter,et al.  Robust Recursive Estimation in the Presence of Heavy-Tailed Observation Noise , 1994 .